Question

I’m getting some very strange behaviour with HttpWebRequest I hope someone can help me with. I have a console app which does some aggregation work by by using the HttpWebRequest object to retrieve the contents of a target website. Due to the nature of the requirement the app is multithreaded and attempts to make anywhere between 10 and 30 simultaneous connections (I’ve been experimenting with a range of values). The actual web request is structured as follows:

var req = (HttpWebRequest)WebRequest.Create(url);
WebResponse resp = req.GetResponse();
Stream s = resp.GetResponseStream();
var sr = new StreamReader(s, Encoding.ASCII);
string doc = sr.ReadToEnd();
sr.Close();
resp.Close();
return doc;

Anyway, the strange behaviour is that under normal circumstances the app is achieving around 120 requests per minute but if I open up Fiddler it jumps to about 600. Using Windows 7 Resource Monitor I can see the network activity increase accordingly. The TCP connections for the console process now list the remote address as “IPv4 loopback” rather than the target server IP address (expected). I did wonder about the max number of simultaneous HTTP requests allowed by the machine but changing this in the registry does not seem to make a difference.

So the question is; what is it about running Fiddler which suddenly increases the throughput five-fold and how can I achieve this natively on the machine without needing to launch another tool?

Thanks!

Answer 1

Looks like I’ve now been able to get the throughput right up (to double that I was getting with Fiddler open actually) by setting the max connections in the App.config:

<system.net>
  <connectionManagement>
    <add address="*" maxconnection="30" />
  </connectionManagement>
</system.net>

Very happy with the result but am still a little mystified as to why having Fiddler open changed the results so dramatically.