Question

I have been using this link to learn about multiprocessing, but I’m stuck on the second example:

import multiprocessing
import time

class Consumer(multiprocessing.Process):

    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue

    def run(self):
        proc_name = self.name
        while True:
            next_task = self.task_queue.get()
            if next_task is None:
                # Poison pill means we should exit
                print '%s: Exiting' % proc_name
                break
            print '%s: %s' % (proc_name, next_task)
            answer = next_task()
            self.result_queue.put(answer)
        return


class Task(object):
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def __call__(self):
        time.sleep(0.1) # pretend to take some time to do our work
        return '%s * %s = %s' % (self.a, self.b, self.a * self.b)
    def __str__(self):
        return '%s * %s' % (self.a, self.b)


if __name__ == '__main__':
    # Establish communication queues
    tasks = multiprocessing.Queue()
    results = multiprocessing.Queue()

    # Start consumers
    num_consumers = multiprocessing.cpu_count() * 2
    print 'Creating %d consumers' % num_consumers
    consumers = [ Consumer(tasks, results)
                  for i in xrange(num_consumers) ]
    for w in consumers:
        w.start()

    # Enqueue jobs
    num_jobs = 10
    for i in xrange(num_jobs):
        tasks.put(Task(i, i))

    # Add a poison pill for each consumer
    for i in xrange(num_consumers):
        tasks.put(None)

    # Start printing results
    while num_jobs:
        result = results.get()
        print 'Result:', result
        num_jobs -= 1

First, could someone please explain exactly what multiprocessing.Process.__init__(self) does? Also I’m not entirely sure how the queue works, and I’m confused how the run method in the Consumer class is executed even though it is never called (explicitly at least…)

If someone could help me walk through the example to get the given output, it would be greatly appreciated.

Answer 1

When an object of a class is created, its __init__() method gets called automatically to initialize it. In the second example, class Consumer is defined as inheriting from multiprocess.Process. The first thing Consumer does in its initialization is to initialize its base Process, to ensure it’s ready to run. Unlike many OO languages, base class __init__() functions aren’t automatically invoked in Python.

The run() method is automatically called in the new process when start() is called on the Consumer object in the main process. That’s in the multiprocessing docs.

The Queue is how multiple processes communicate. In general, separate processes can’t see each other’s data, but Queue lets them pass messages back and forth. In this example, the main process is putting several tasks into a queue, and each of the other processes will pull a task from that queue, carry it out, and send the result back on another queue.