Web request with HTTP authentication from Powershell

Web requests can be made from Powershell with HTTP authentication in the following 2 ways:

1. Using credential objects
$username = "foo"
$password = "bar" | ConvertTo-SecureString -asPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential($username,$password)
$res = Invoke-WebRequest http://localhost:3000 -Credential $cred
$res.Content

2. Adding appropriate header
$acctname = "foo"
$password = "bar"
$params = @{uri = 'http://localhost:3000';
                   Method = 'Get'; #(or POST, or whatever)
                   Headers = @{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($acctname):$($password)"));
           } #end headers hash table
   } #end $params hash table
$var = curl @params

Curl above is same as Invoke-WebRequest because on Windows, by default curl is just an alias to Invoke-WebRequest.


A JSON request with a body can be made as follows:

$res = Invoke-WebRequest http://localhost:3000/stores -Credential $cred -Method POST -Headers @{"Content-Type"="application/json"} -Body (ConvertTo-Json @{"Key"="Value"})

Bypassing SSL in WebRequests

Using C#, there are two prevalent ways of making HTTP requests:
- using HttpWebRequest
- using HttpClient

When you have a cancellation token that you want to use, only the later provides the facility for that. So, there are scenarios for each method to be adopted is what I am getting at.

While accessing HTTPS sites whose SSL certificates are not trusted, we manually allow access in case of the browser. The same can b achieved via code using the following methods.

1. Setting ServerCertificateValidationCallback for the request.
This method only for the first method of making HTTP requests.

// provide a custom callback
request.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(SSLValidationDelegate);

// define the callback to allow always
private static bool SSLValidationDelegate(Object o, X509Certificate cert, X509Chain chain, System.Net.Security.SslPolicyErrors errors){
    return true;
}

2. Setting ServerCertificateValidationCallback for the ServicePointManager class.
This method essentially does the same but instead of doing it at the request level it does that for all.

ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
HttpClient client = new HttpClient();
CancellationTokenSource cts = new CancellationTokenSource();
cts.CancelAfter(1000);
string url = "https://localhost/some/path";
HttpResponseMessage response = client.GetAsync(url, cts.Token).Result;


So when a cancellation needs to be used,we have to take the second route.

HolidayIQ share feature broken

The HolidayIQ app did not have share facility before so you could only read their blogs when you get a push notification but not share it with somebody. Recently, I was glad to see the share feature in their app. However, when I tried using it by sharing on WhatsApp, it seems they messed up again. The shared URL is a shortened URL so the problem is not quite apparent on first sight but when a person opens the URL in a browser on a phone that does not have the app, the URL does not work with a message that the URL was not understood (This message was from Firefox. Other browsers will have similar issues.). The problem is that the URL specifies incorrect protocol. Instead of http:// the shortened URL expands to one starting with holiday:// With phones that have the app, this URL is opened in the HolidayIQ app and causes no issues. However, I think they should use a http:// URL while sharing.

Disable compression in Net::HTTP

Recently while upgrading to Ruby 2, I found a JSON API call was failing intermittently. The code was failing when I was trying to parse the response using JSON.parse. Investigating further, I found that the call was successful; but for some calls the response was different than the response for others. Looking at the response, I found it was not JSON at all. It looked like the following.

\x8B\x00\x00\x8C...

I had no clue as to what to make out of such a response. Reading up on it, I found out that the above might be a compressed response. It appeared that the server had the liberty of choosing whether to compress the response or not and whenever the response was uncompressed, my code was working fine. Looking into improvements introduced in Ruby 2, I found that Net::HTTP now automatically requests gzip and deflate compression by default. All I needed to do was to stop doing that and ask for uncompressed response. As my response was rather small, it would hardly matter. To request uncompressed response, I just needed to add the following header to my requests.

'Accept-Encoding' => 'identity'

Remove X-Powered-By header in Nginx server

It is often desired to remove the 'X-Powered-By' from the HTTP headers. When using Nginx, it can be using a simple setting. The setting has to be in http section and it takes the following form.

<module_name>_hide_header X-Powered-By;

For example, if it is a FastCGI server the configuration setting will be as follows:

fastcgi_hide_header X-Powered-By;

If you are using Nginx as a load balancer, the configuration setting will be as follows:

proxy_hide_header X-Powered-By;

Unfortunately this configuration setting is not available yet in Passenger because Passenger is a thord party module and they have not implemented it yet.