Question

I don’t understand why Pipes are said unsafe when there are multiple senders and receivers.

How the following code can be turned into code using Queues if this is the case ? Queues don’t throw EOFError when closed, so my processes can’t stop. Should I send endlessly ‘Poison’ messages to tell them to stop (this way, i’m sure all my processes receive at least one poison) ?

I would like to keep the pipe p1 open until I decide otherwise (here it’s when I have sent the 10 messages).

---

from multiprocessing import Pipe, Process
from random import randint, random
from time import sleep

def job(name, p_in, p_out):
    print(name + ' starting')
    nb_msg = 0
    try:
        while True:
            x = p_in.recv()
            print(name + ' receives ' + x)
            nb_msg = nb_msg + 1
            p_out.send(x)
            sleep(random())
    except EOFError:
        pass
    print(name + ' ending ... ' + str(nb_msg) + ' message(s)')

if __name__ == '__main__':
    p1_in, p1_out = Pipe()
    p2_in, p2_out = Pipe()

    proc = []

    for i in range(3):
        p = Process(target=job, args=(str(i), p1_out, p2_in))
        p.start()
        proc.append(p)

    for x in range(10):
        p1_in.send(chr(97+x))
    p1_in.close()
    for p in proc:
        p.join()
    p1_out.close()
    p2_in.close()

    try:
        while True:
            print(p2_out.recv())
    except EOFError:
        pass

    p2_out.close()

Answer 1

Essentially, the problem is that Pipe is a thin wrapper around a platform-defined pipe object. recv simply repeatedly receives a buffer of bytes until a complete Python object is obtained. If two threads or processes use recv on the same pipe, the reads may interleave, leaving each process with half a pickled object and thus corrupting the data. Queues do proper synchronization between processes, at the expense of more complexity.

As the multiprocessing documentation puts it:

Note that data in a pipe may become corrupted if two processes (or threads) try to read from or write to the same end of the pipe at the same time. Of course there is no risk of corruption from processes using different ends of the pipe at the same time.

You don’t have to endlessly send poison pills; one per worker is all you need. Each worker picks up exactly one poison pill before exiting, so there’s no danger that a worker will somehow miss the message.

You should also consider using multiprocessing.Pool instead of reimplementing the “worker process” model — Pool has a lot of methods which make distributing work across multiple threads very easy.