Question

I have around 1000 processes which should be executed independently. I have 8 cores. Is it possible that I start all the threads and python will take care of managing them or should I keep track of which thread has been executed and then start other one.

In case of the second option is the right way, could anybody please explain the simplest way of doing it?

Thanks a lot

Answer 1

What you describe sounds like an ideal application for a thread pool. This involves a relatively small number of workers that take tasks off a queue and process them.

In Python, it is very hard to achieve true parallelism through threading. Fortunately, there is an alternative: the multiprocessing module. It even includes some facilities for managing pools of workers.