Question

Background: I’m using .NET 4.0 and the new Task class. Using the code found at http://msdn.microsoft.com/en-us/library/ee789351.aspx I’ve implemented a Task Scheduler that limits the number of concurrent threads executing. I’ve also modified the code to use the BlockingCollection instead of LinkedList for my lists of tasks so I can limit how large the list of pending tasks gets. Each of my tasks potentially spawn other tasks. So, task A can spawn task B, C, and D.

Problem: Task A isn’t technically complete until it’s added task B, C, and D to the queue, but if the queue is full, A blocks. So, how do I have a limited task queue with self expanding tasks?

The reason I need to limit the size of the queue is it will otherwise explode my memory use. A single task can spawn thousands of other tasks. 10-15 tasks each queuing up thousands more…you get the picture.

Suggestions would be appreciated!

Thanks,
Dan

Answer 1

how do I have a limited task queue with self expanding tasks

I would say you don’t; that is, your requirements are in conflict with each other. If you must have a hard limit on the number of pending tasks, then you must accept that an attempt to pend a new task can either block (as now) or fail.

If “a single task can spawn thousands of other tasks” then you are necessarily going to have the possibility of a large amount of pending work. The purpose of the task queue is to act as a place to hold pending work. However, since (I would hope) most tasks will not pend thousands of new tasks, over time the amount of pending work will diminish, eventually to nothing.

In a sense, one of the points of having a queue of pending tasks is precisely that new pends can be done without regard to currently available processing time. ‘Thousands’ of items in the wait queue should not be a problem, as far as memory use goes. Millions maybe, but even then – have you profiled and demonstrated this to be a problem?