Question

I’m somewhat confused about multiprocessing. I’m a 3 years Python programmer, however never really needed to perform tasks in parallel (not just asynchronously). But what I know, or thought to know, is that when using the multiprocessing module in Python to achieve “real parallelism”, new python.exe processes are spawned!

A 3D software like Cinema 4D for example uses the power of all CPUs available to render a 3D scene. But I don’t see multiple Cinema 4D.exe processes in the Task Manager.

1. Am I correct with the statement above, that multiple Python processes are spawned when using the multiprocessing module?
2. If so, why is it, and how can the C++ application use all CPUs without multiple processes?

Answer 1

It’s also possible to use multiple CPUs by running multiple threads in the same process. That’s just not what the Python multiprocessing module does.

There is a threading module in Python. Unfortunately in CPython, threads aren’t as useful as you might think because they all contend on the so-called “Global Interpreter Lock”. So they’re a lot less parallel in Python than they are in many other languages. If you use threads, you need to worry about what operations in your code will really be parallel. If you use processes you don’t (although you might worry about other things, like sharing data).

I don’t know whether not or alternative implementations of Python have the same issue with the GIL. But unless you’re writing code specifically for Jython/IronPython/whatever, the limitations of CPython apply to your program…