Question

I am creating a small application which will perform say 4 different, time consuming tasks such that the output of first is the input of second and so on.
At every task level, the output is appended to a list and the next task pops, operates and appends its output to its output list and so on…
The way I thought I would get the task done is by having multiple threads on each of those 4 tasks.

Coming to the question, is there any way using which I can have my application spawn threads at each task level depending upon the number of tasks in its input queue?

Say the input list of the second task is empty in the beginning so number threads is zero however if one task is there, single thread is spawned, two for two etc … And of course have an upper limit on the number of threads say 10, so that if the length of the input list goes as high as 100, number of threads operating stays still at 10.

Please suggest the pythonic way to go about achieving this.

Answer 1

You have successfully invented the Thread Pool. There is builtin support and there are many libraries and examples that will provide this for you, so use one or learn from their code.

from multiprocessing.pool import ThreadPool