Question

Is there something similar to C#’s excellent Parallel.For in Python? I just want to do a calculation like

[simu(c) for c in clusterSizes]

in parallel. What is the simplest way to archive that?

PS: I tried joblib, but in my cases it just starts, starts and starts processes until i have to restart my machine.

Answer 1

In python 3, there is parallel map in concurrent.futures (in standard library). I think it was even backported as a module for python 2.7.
edit http://pypi.python.org/pypi/futures

As noted in other answer, threads won’t help you. Instead you have to use multiple processes.

edit from docs it seems as simple as this:

with concurrent.futures.ProcessPoolExecutor() as executor:
    for result in executor.map(simu, clusterSizes)
        pass # save result