Question

I’m starting a little “job manager”, it will execute some “Job” object implement an “IJob” method. Every IJob finish by returning a potential list of new jobs.

All my jobs are stored in a Queue, which is accessed thread-safely, and when I finish a Job, I add to this queue new queried jobs.

I’m searching how to parallelize this with the TPL.

I’ve several ideas, but none of them are valid due to my constraints, which are:

• I need to limit the number of thread(let’s say to 4), because some request are quering a website, which doesn’t allow more than 4 simultaneous requests(I think I can manage this with a semaphore)
• My list of jobs will change, so I just cannot launch 4 thread, divide the number of jobs by the number of thread, and every thread runs his stack of job.
• Maybe sometimes, if I’ve 4 thread, I’ve only one job more, but I can’t let stop other thread, because maybe the last job I’ve to run will create a lot of more jobs.

Thank you very much!

Answer 1

This is really dynamic task parallelism. Your code loops through the Jobs you have and execute each one. Each job can add new jobs to the using an addMethod and a concurrent queue.

public static void ParallelWhileNotEmpty<T>(
  IEnumerable<T> initialValues, 
  Action<T, Action<T>> body)
{
  var opts = new ParallelOptions { MaxDegreeOfParallelism = 10 };
  var from = new ConcurrentQueue<T>(initialValues);  
  while (!from.IsEmpty)
  { 
    var to = new ConcurrentQueue<T>();
    Action<T> addMethod = to.Enqueue; 
    Parallel.ForEach(from, opts. body(v, addMethod));        
    from = to;
  }
}

This way the “loop” is open ended and will continue until you run out of work. Obviously your real application would account for duplicate URLs and not add them etc. But this allows your to app to dynamically add work. You can use the ParallelOptions to limit concurrency or you can write a scheduler.

For more information on dynamic task parallelism see

http://msdn.microsoft.com/en-us/library/ff963551.aspx

For the full code for the example see

http://parallelpatterns.codeplex.com/SourceControl/changeset/view/54510#795590

Both of the above discuss other alternative variations on this theme.

If you want a custom scheduler to limit the degree of parallelism see the example on MSDN

http://msdn.microsoft.com/en-us/library/ee789351.aspx