Question

I was trying to work on a component that uses the multiprocessing module in PyDev, but was having problems where running the code was spawning 100s of python processes in the OS and killing my machine. I had my code call run() instead of start(), and it worked fine (except, of course, it was all single threaded since start() is what actually spawns the processes), so that suggests to me that my code was at least not doing anything insane like infinitely generating processes or something. Further, I put together a little test code to see what was up:

from multiprocessing import Process
import time

def create_processes(num_processes, method, *args):

    processes = []
    for i in range(num_processes):
        processes.append(Process(target=method, args=args))
    return processes

def start_all(processes, stagger):

    for process in processes:
        process.start()
        if stagger:
            time.sleep(stagger)

processes = create_processes(2, time.sleep, 4)
start_all(processes, 0)

When I run this in the cli, it works fine; it spawns 2 new processes that end after the timeout. However, when I run it in PyDev, it generates ~600 processes (or at least that’s how many task manager shows before my machine becomes unresponsive). I did a little debugging, and it seems to be blowing up in Process.start() on the line self._popen = Popen(self).

I did a little searching around, but couldn’t really find anything that seemed relevant. Is the CLI automatically join()ing all start()ed processes to some separate manager to keep the reins on? Do I need a thread in my code to join() all my processes to? Is PyDev jacking around with things to try to maintain logging and control? Is something else going on?

Answer 1

If this is the starting point to the code, you should wrap the final two lines with if __name__ == '__main__' in order to prevent the subprocess from starting their own subprocesses:

if __name__ == '__main__':
    processes = create_processes(2, time.sleep, 4)
    start_all(processes, 0)