Question

I need to use a thread pool in python, and I want to be able to know when at least 1 thead out or “maximum threads allowed” has finished, so I can start it again if I still need to do something.

I has been using something like this:

def doSomethingWith(dataforthread):
    dostuff()
    i = i-1 #thread has finished

i = 0
poolSize = 5
threads = []
data = #array of data
while len(data):
    while True:
        if i<poolSize: #if started threads is < poolSize start new thread
            dataforthread = data.pop(0)
            i = i+1
            thread = doSomethingWith(dataforthread)
            thread.start()
            threads.append(thread)
        else:
            break
    for t in threads: #wait for ALL threads (I ONLY WANT TO WAIT FOR 1 [any])
        t.join()

As I understand, my code opens 5 threads, and then waits for all the threads to finish before starting new threads, until data is consumed. But what I really want to do is start a new thread as soon as one of the threads finish and the pool has an “available spot” for a new thread.

I have been reading this, but I think that would have the same issue than my code (not sure, im new to python but by looking at joinAll() it looks like that).

Does someone has an example to do what I am trying to achieve?

I mean detecting as soon as i is < than poolSize, launching new threads until i=poolSize and do that until data is consumed.

Answer 1

As the article author mentions, and @getekha highlights, thread pools in Python don’t accomplish exactly the same thing as they do in other languages. If you need parallelism, you should look into the multiprocessing module. Among other things, it has these handy Queue and Pool constructs. Also, there’s an accepted PEP for “futures” that you’ll probably want to monitor.