Question

My program has 2 parts divided into the core and downloader. The core handles all the app logic while the downloader just downloads urls. Right now, I am trying to use the python multiprocessing module to accomplish the task of the core as a process and the downloader as a process.

The first problem I noticed was that if I spawn the downloader process from the core process so that the downloader is the child process and the core is the parent, the core process(parent) is blocked until the child is finished. I do not want this behaivor though. I would like to have a core process and a downloader process that are both able to execute their code and communicate between each other.

example

...
def main():
    jobQueue = Queue()
    jobQueue.put("http://google.com)
    d = Downloader(jobQueue)
    p = Process(target=d.start())
    p.start()

if __name__ == '__main__':
    freeze_support()
    main()

where Downloader’s start() just takes the url out of the queue and downloads it.

In order to have the 2 processes unblocked, would I need to create 2 processes from the parent process and then share something between them?

Answer 1

On the p = Process(target=d.start()) line, you’re calling d.start(). Remove the parens, like below:

...
def main():
    jobQueue = Queue()
    jobQueue.put("http://google.com")
    d = Downloader(jobQueue)
    p = Process(target=d.start)
    p.start()

if __name__ == '__main__':
    freeze_support()
    main()