Question

I have a for loop running through 500.000ish list. For each of these it is queueing up a SmartThreadPool job.

lines.Length below contains 500.000ish items.

My problem is that i get memory issues when queueing them all at once.. So i though id write a logic to prevent this:

int activeThreads = _smartThreadPool2.ActiveThreads;
if (activeThreads < maxThreads) 
{
    int iia = 0;
    for (int i = 0; i < lines.Length; i++)
    {
        if (doNotUseAdditive.Checked == true)
        {
            foreach (string engine in _checkedEngines) // Grab selected engines
            {
                query = lines[i];

                _smartThreadPool2.QueueWorkItem(
                new Amib.Threading.Func<string, string, int, int, int>(scrape),
                query, engine, iia, useProxies);

                iia++;
            }
        }
    }
}
else
{
    // Wait
    wait.WaitOne();
}

The problem is that i cannot run that if statement inside my for loop, because when i come back to it, it will not remember where it was inside the loop.

I’m using a:

ManualResetEvent wait = new ManualResetEvent(false);  //global variable

To “Pause/Resume”

I need to somehow pause the loop after X threads are used and then when threads are available return and continue the loop.

Any ideas?

Answer 1

I don’t think that process every item in list in separate thread is a good idea. Even using custom thread pool can be really error-prone (and you examples proves my opinion).

First of all you should determine number of working threads correctly. It seems that you’re dealing with computation intensive operations (so called CPU Bound operations) and you should use number of working threads equals to number of logical processors.

Than you can use Parallel LINQ to split all your working set for appropriate amount of chunks and process those chunks in parallel.

Joe Albahari has a great series of posts about this topic: Threading in C#. Part 5. Parallel Programming.

Here is a pseudocode of using PLINQ:

lines
  .AsParallel()
  .WithDegreeOfParallelism(YourNumberOfProcessors)
  .Select(e => ProcessYourData(e));