Question

I have problem with python multithreaded Queues. I have this script, where producer take elements from input queue, produces some elements and puts them to output queue, and consumer takes element from output queue and just prints them:

import threading
import Queue

class Producer(threading.Thread):
    def __init__(self, iq, oq):
        threading.Thread.__init__(self)
        self.iq = iq
        self.oq = oq

    def produce(self, e):
        self.oq.put(e*2)
        self.oq.task_done()
        print "Producer %s produced %d and put it to output Queue"%(self.getName(), e*2)

    def run(self):
        while 1:
            e = self.iq.get()
            self.iq.task_done()
            print "Get %d from input Queue"%(e)
            self.produce(e)


class Consumer(threading.Thread):
    def __init__(self, oq):
        threading.Thread.__init__(self)
        self.oq = oq

    def run(self):
        while 1:
            e = self.oq.get()
            self.oq.task_done()
            print "Consumer get %d from output queue and consumed"%e

iq = Queue.Queue()
oq = Queue.Queue()

for i in xrange(2):
    iq.put((i+1)*10)

for i in xrange(2):
    t1 = Producer(iq, oq)
    t1.setDaemon(True)
    t1.start()

    t2 = Consumer(oq)
    t2.setDaemon(True)
    t2.start()

iq.join()
oq.join()

But, every time I run it, it works different(gives exception, or consumer does not do any job). I think the problem is in task_done() command, can anyone explain me where the bug is?

I have modified Consumer class:

class Consumer(threading.Thread):
    def __init__(self, oq):
        threading.Thread.__init__(self)
        self.oq = oq

    def run(self):
        while 1:
            e = self.oq.get()
            self.oq.task_done()
            print "Consumer get %d from output queue and consumed"%e
            page = urllib2.urlopen("http://www.ifconfig.me/ip")
            print page

Now consumer after each task_done() command should connect to web site (it takes some time), but it does not, instead if execution time of code after task_done() is small, it runs but if it is long it does not run! Why? Can anyone explain me this issue? If I put everything before task_done() command then I will block queue from other threads which is stupid enough. Or is there anything I am missing about multithreading in python?

Answer 1

From the Queue docs:

Queue.task_done() Indicate that a formerly enqueued task is complete.
Used by queue consumer threads. For each get() used to fetch a task, a
subsequent call to task_done() tells the queue that the processing on
the task is complete.

If a join() is currently blocking, it will resume when all items have
been processed (meaning that a task_done() call was received for every
item that had been put() into the queue)

For example in your code you do the following in your Producer class:

def produce(self, e):
    self.oq.put(e*2)
    self.oq.task_done()
    print "Producer %s produced %d and put it to output Queue"%(self.getName(), e*2)

You shouldn’t do self.oq.task_done() here, since you haven’t used oq.get().

I am not sure this is the only problem though.

EDIT:

For your other problem, you’re using iq.join() and oq.join() at the end, this leads your main thread to exit before the other threads print the retrieved pages, and since you’re creating your threads as Daemons, your Python application exits without waiting for them to finish executing. (Remember that Queue.join() depends on Queue.task_done())

Now you’re saying “If I put everything before task_done() command then I will block queue from other threads”. I can’t see what you mean, this will only block your Consumer thread, but you can always create more Consumer threads which won’t be blocked by each other.