Question

I am using the fantastic SmartThreadPool to queue up a bunch of parsing tasks.

So basically what happens is that when the users clicks the button Start, the program will loop through different lists and based on those lists it will start different jobs (note: this can be 500.000 jobs).

I am queueing up like this:

 private void button1_Click(object sender, EventArgs e)
 {
     button1.Enabled = false;
     button2.Enabled = true;

     stpStartInfo.MaxWorkerThreads = Convert.ToInt32(parserThreadCount.Value);
     stpStartInfo.MinWorkerThreads = 1;
     _smartThreadPool2 = new Smart.SmartThreadPool(stpStartInfo);

     ........

     foreach (string engine in _checkedEngines)
     {
         query = lines[i];

         _smartThreadPool2.QueueWorkItem(
         new Amib.Threading.Func<string, string, int, int, int>(scrapeFunction),
         query, engine, iia, useProxies);

         iia++;
     }
}

So the problem is that the user has to wait (interface almost hangs) when queueing up like 200.000+ threads..

The ideal solution would be for a way to let the threads that have already queued up start, and then queue the rest in the background.. But have no idea how i could accomplish this..

Any ideas? 🙂

Answer 1

You can use one of the threads to queue the jobs. Just move the foreach to a separate method and queue it in your main thread. That will be enough to do the queuing in parallel and when it’s done, its thread will be recycled as well by the SmartThreadPool and used for the queued jobs.