Those who work with the. NET framework and occasionally used the HttpWebRequest class, may have stumbled about the phenomenon that it seems to be quite slow in some cases. Especially if you use HttpWebRequest in combination with threading to get responses as quick as possible, the HttpWebRequest class quickly becomes a party pooper.
Initially you might assume that the problems rely on the Internet connection, but at least after a speed comparison with wget or curl, you should recognize that the “problem” must be at the. NET implementation.
1. Increase DefaultConnectionLimit
All instances of the HttpWebRequest class are using some properties of the ServicePointManager. One of these porperties of the ServicePointManager is the “DefaultConnectionLimit”. By default, this limit is set to 2, which means that there will be never more than 2 simultaneous connections.
Especially if you want to check multiple requests simultaneously (e.g. by threading) , this becomes an absolute brake. In this case, the DefaultConnectionLimit should be scaled up wisely and according to the capacity of the available network connection.
ServicePointManager.DefaultConnectionLimit = connectionLimit;
To find the optimal value, you should make multiple test runs with different values.
2. Make proxy settings correctly
Another bottle-neck may be the proxy server settings. If you search the internet for problems with HttpWebRequest, you will find frequently a problem description, which says, that only the first query takes a long time and all further queries go relatively quickly. The reason for that is, that the first query, after instantiation of the the HttpWebRequest, is looking for the system-wide proxy settings. This search progress can easily take 2-8 seconds. There are two solutions for that.
No proxy needed
If no proxy is needed, it can be explicitly set to null, so it will not look for the system-wide default proxy.
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("http://mywebsite.com"); req.Proxy = null;
It will need a proxy
If a proxy server is required and it’s settings are known (what should be the case, if you program in an environment where a proxy is needed) then this settings can be set manually. So the slow search progress for the proxy settings can also be bypassed.
string proxyUrl = "proxy.myproxy.com"; int proxyPort = 8080; WebProxy myProxy = new WebProxy(proxyUrl, proxyPort); HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("http://mywebsite.com"); req.Proxy = myProxy;
3. Deactivate Expect100Continue
The last tip may also bring you a couple of seconds. More specifically – about 1/3 second per query. Depending on the number of queries this can be a good enhancement, too. In the default configuration a Continue(100) status is sent before every query. If the server is ready/willing to process our query, our client receives the response from the server and sends the actual request.
If we assume that we know the server and know that he will accept our query, we can disable the Continue(100) requests. In addition, there is still some server-software that does not get along with Continue(100) hits. More information about that can be found on this webpage.
ServicePointManager.Expect100Continue = false;
That’s it. Should you know other ways to accelerate HttpWebRequests, so just write me a comment.