Home/ Questions/Q 9288859
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-18T20:00:40+00:00

I have a script which does a lot of HttpRequsts in a while loop

  • 0

I have a script which does a lot of HttpRequsts in a while loop as follows. It is run in a BackgroundWorker.

CookieContainer cookieJar = new CookieContainer();
string searchURL = "http://blablabla.com/";
bool shouldRun = true; //this is set elsewhere

while (shouldRun)
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(searchURL);
    request.ContentType = "application/json";
    request.Method = "POST";
    request.ServicePoint.Expect100Continue = false;
    request.CookieContainer = cookieJar;
    request.Headers.Add("X-HTTP-Method-Override: GET");
    request.Timeout = 1000;

    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    // Get the stream containing content returned by the server.
    Stream responseStream = response.GetResponseStream();
    // Open the stream using a StreamReader for easy access.
    StreamReader reader = new StreamReader(responseStream);
    // Read the content.
    string responseFromServer = reader.ReadToEnd();

    JObject o = JObject.Parse(responseFromServer);

    //Do something with the JObject
}

Running the script takes up about 10 % of the CPU (Intel i5 @ 2.5GHz), and I would like to explore other approaches to see if it would be able to reduce the CPU load without reducing the number of requests made per second (current rate is around 3 to 4 requests/s) or whether it would be possible to increase the number of requests per second.

I’ve been looking at asynchronous requests, but would it be able to give me one of the two improvements mentioned above?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Doing an HTTP request isn’t very CPU intensive, you could do 20+ of these without using 10% of the cpu. What is cpu intensive is what you do with that data. Parsing it (as well as whatever you’re doing with that parsed data) is what’s taking up most of your cpu. You should look into other ways of dealing with that data to save processing power. For example maybe instead of processing the data as you get it, you could save that data and once a minute process all that saved data at once with some script.

    • 0