I’ve got a queue (actually 3 different queues but I don’t believe that will make a difference). The queue contains discrete units of work.
I have a QueueManager which spawns a new thread and returns. The new thread looks at the queue, gets a piece of work and then spawns a new thread to process it.
I obviously want a limit on the number of concurrent threads and don’t see any reason not to reuse a thread once it’s finished the job.
Some of the jobs will take a while to run (minutes/hours) and I’ve read that the ThreadPool is not a good option for anything that takes more than a few seconds.
Effectively, I want to implement the ThreadPool but with a little more control.
So… What’s the standard, best-practice way to achieve this?
It seems I need to (assuming the queue length > MaxThreadCount)
- Create new jobs until I hit
MaxThreadCount - Wait until any thread completes
- Get the next job from the queue and assign it to the thread
- Repeat
- If the queue empties, sleep for a length of time and then re-check
(The queue in this case is in a database so no event when items are added – I assume I’ll just have to poll)
A lot of the jobs involve retrieving pages from remote sites/APIs, often repeatedly. As such, I believe I’d benefit from having considerably more threads than cores (as most of them will be waiting for the network). As such, I don’t believe WaitHandle is appropriate as it has a 64-Thread limit.
This is obviously a pretty common pattern so I assume there must be a well-established way of implementing it?
If someone could point me at a good example, the would be excellent.
Use the TPL (Task Parallel Library). It’s designed for exactly this purpose.
http://msdn.microsoft.com/en-us/library/dd460717.aspx
Specifically, it sounds like you should create a TaskFactory with a TaskScheduler instance whose MaximumConcurrencyLevel has been set to the number of threads that you want. Then instead of putting your work items in a queue, call TaskFactory.StartNew. It will handle the queueing of work for you.