Question

trying to get to the bottom of this!

i have a very basic app that is using httpwebrequests to login, navigate to a page and then grab the html of that page. it then preforms another webrequest to a third page every 5 mins in a loop.

its all working fine and is single threaded (and fairly old), however circumstances have changed and i now need to run multiple instances of this app closely together (i have a .bat starting the app every 2seconds as a temporary measure until i am able to code a new multithreaded solution).

when the first instances of the app start everything is fine, first request is completed in ~2seconds. second one in about 3seconds.

however as more and more instances of this app are run concurrently (>100) something strange starts to happen.

the first web request still takes ~2 seconds, however the second request gets delayed much more >1min up to the point of timeout. i cant seem to think why this is. the second page is larger than the first, but nothing out of the ordinary that would take >1min to download.

The internet connection and hardware of this server is more than capable of handling these requests.

 CookieContainer myContainer = new CookieContainer();
        // first request is https
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(https://mysite.com/urlone);                                
        request.CookieContainer = myContainer;
        request.Proxy = proxy;
        Console.WriteLine(System.DateTime.Now.ToLongTimeString() + "  " + "Starting login request");
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        Stream resStream = response.GetResponseStream();

        string tempString = null;
        int count = 0;

        do
        {
            // fill the buffer with data
            count = resStream.Read(buf, 0, buf.Length);

            // make sure we read some data
            if (count != 0)
            {
                // translate from bytes to ASCII text
                tempString = Encoding.ASCII.GetString(buf, 0, count);

                // continue building the string
                sb.Append(tempString);
            }
        }
        while (count > 0); // any more data to read?
        sb.Clear();
       response.Close();
       resStream.Close();
       string output6;



        Console.WriteLine(System.DateTime.Now.ToLongTimeString() + "  " + "login request comeplete");
        HttpWebRequest request6 = (HttpWebRequest)WebRequest.Create(@"http://mysite.com/page2");
        request6.CookieContainer = myContainer;
        response = (HttpWebResponse)request6.GetResponse();
        resStream = response.GetResponseStream();
        tempString = null;
        count = 0;

        do
        {
            count = resStream.Read(buf, 0, buf.Length);


            if (count != 0)
            {
                tempString = Encoding.ASCII.GetString(buf, 0, count);

                sb.Append(tempString);
            }
        }

        while (count > 0);
        output6 = sb.ToString();
        sb.Clear();
        response.Close();
        resStream.Close();

Any ideas? Im not very advanced with http web requests so if someone could check i haven’t made any silly code mistakes above id appreciate it. Im at a loss as to what other information i may need to include here, if i have missed anything out please tell me and i will do my best to provide.
Thanks in advance.

EDIT 1:

I used fiddler to find out the source of the issue. It looks like the issue lies with the application (or windows) not sending the requests for some reason – the physical request actually takes < 1second according to fiddler.

Answer 1

Check out a few things

1. ServicePointManager.DefaultConnectionLimit : if you are planning to open more then 100 connection the set this value to something like 200-300.
2. If possible use HttpWebRequest.KeepALive = true