Question

Aside from the ease of use of the multiprocessing module when it comes to hooking up processes with communication resources, are there any other differences between spawning multiple processes using multiprocessing compared to using subprocess to launch separate Python VMs ?

Answer 1

On Posix platforms, multiprocessing primitives essentially wrap an os.fork(). What this means is that at point you spawn a process in multiprocessing, the code already imported/initialized remains so in the child process.

This can be a boon if you have a lot of things to initialize and then each subprocess essentially performs operations on (copies of) those initialized objects, but not all that helpful if the thing you run in the subprocess is completely unrelated.

There are also implications for resources such as file-handles, sockets, etc with multiprocessing on a unix-like platform.

Meanwhile, when using subprocess, you are creating an entirely new program/interpreter each time you Popen a new process. This means there can be less shared memory between them, but it also means you can Popen into a completely separate program, or a new entry-point into the same program.

On Windows, the differences are less between multiprocessing and subprocess, because windows does not provide fork().