Question

I modified an example with threads and got the example with multiprocessing included below. My problem is that ProduceToQueue ran fine, however ConsumeFromQueue did not finish. Why? This is one of my first multiprocessing programs, so be patient.

from Queue import Queue
import multiprocessing
import random
import time


class ProduceToQueue(multiprocessing.Process):
    def __init__(self, queue):
        multiprocessing.Process.__init__(self)
        self.shared_queue = queue

    def run(self):
        for i in range(11, 21):
            time.sleep(random.randrange(5))
            print "%s adding %s to queue" % (self.name, i)
            self.shared_queue.put(i)

        print self.name, "finished producing values"
        print "Terminating", self.name


class ConsumeFromQueue(multiprocessing.Process):
    def __init__(self, queue):
        multiprocessing.Process.__init__(self)
        self.shared_queue = queue

    def run(self):
        value = 0
        current = 10
        for i in range(10):
            time.sleep(random.randrange(3))
            print "%s attempting to read %s..." % (self.name, current + 1)
            current = self.shared_queue.get()
            print "%s read %s" % (self.name, current)
            value += current

            print "%s retrieved values totaling: %d" % (self.name, value)
            print "Terminating", self.name


queue = Queue()
producer = ProduceToQueue(queue)
consumer = ConsumeFromQueue(queue)

producer.start()
consumer.start()

producer.join()
consumer.join()

Answer 1

If you’re using multiprocessing instead of threading, you should be using the Queue class from multiprocessing since it’s the one that shares properly amongst processes. See the Python docs on the multithreading module for details.

The standard Queue class (from Queue) will effectively be split into two separate queues whenever you create a child process. That means, in your situation, you have the queue in the main process (which nobody is touching), a different queue in the producer (which is being written to) and yet another queue in the consumer (which is being read from).

And there’s no connection whatsoever between those three.

The quickest fix is to simply change:

from Queue import Queue

into:

from multiprocessing import Queue

so that your queues are of the correct type to work between processes rather than just threads. Or just get rid of that from line altogether and use:

queue = multiprocessing.Queue()

In addition (though I realise this is just play code), it’s very unusual to have a sleep operation in the consumer, you would usually want to just keep that ready to roll for when a item appears on the queue.