Question

I have a simply foreach loop that limits itself based on while loops and a static int. If I dont limit it, my CPU stays under 10% if i limit it my CPU goes up to 99/100%. How do I safely limit the number of calls to a class within a Paralell.Foreach?

static int ActiveThreads { get; set; }
static int TotalThreads { get; set; }

var options = new ParallelOptions();
options.MaxDegreeOfParallelism = 1;
Parallel.ForEach(urlTable.AsEnumerable(),options,drow =>
{
    using (var WCC = new MasterCrawlerClass())
    {
        while (TotalThreads <= urlTable.Rows.Count)
        {
            if (ActiveThreads <= 9)
            {
                Console.WriteLine("Active Thread #: " + ActiveThreads);
                ActiveThreads++;
                WCC.MasterCrawlBegin(drow);
                TotalThreads++;
                Console.WriteLine("Done Crawling a datarow");
                ActiveThreads--;
            }
        }
    }
});

I need to limit it, and yes I understand Max Parallelism has it’s own limit, however, my switch gets bogged down before the CPU in the server hits that limit.

Answer 1

Two things :

1) You don’t seem to be using your ParallelOptions() that you created in this example.

2) You can use a Semaphore if for some reason you don’t want to use the ParallelOptions.

 Semaphore sm = new Semaphore(0, 9);

 // increment semaphore or block if = 9
 // this will block gracefully without constantly checking for `ActiveThreads <= 9`
 sm.WaitOne();

 // decrement semaphore
 sm.Release();