Question

My requirement is similar to Multiple producers, single consumer
except i need it in python

I have created an application that spawns 5 concurrent processes (I am using multiprocessing library).These 5 process are independently producing output in dict format.

Earlier I was printing the output to console, but now would like to output it to a file.

I am looking for a pattern where all my 5 producers write to a shared queue that supports concurrent writes.

And a single consumer process that too has access to this queue and consumes the data from it, with the ability to wait if there is no data to write and terminate when producers are done with their task.

Thanks Anuj

Answer 1

since you are already using multiprocess, all you need is the Queue class

and a sample (modified from the Queue docs)

from multiprocessing import Process, Queue

def child(q, url):
    result = my_process(url)
    q.put(result)

if __name__ == '__main__':
    q = Queue()
    urls = [...]
    children = []
    for url in urls:
       p = Process(target=child, args=(q,url))
       p.start()
       children.append(p)
    for p in children:
       p.join()
       print q.get() #or write to file (might not be the answer from this child)

Edit:
For multiple answers from each child replace the last for loop with:

while 0 != multiprocessing.active_children():
    print q.get()