Question

I feel like this should be an easy solution but it’s the end of the day and I’m brain-dead.
I am currently spawning a couple of processes, one process is receiving and storing data to a file. Another is parsing the data and third is waiting for user input to know when to stop the storing of data.

What I need to know how to do is breakout of my while loop. I’d like to not use global variables set by the parent process but if that is required I can do that.
Right now my code looks something like this:

while(packetReceived < totalToReceive):
    data, addr = sock.recvfrom(packetSize)

My thoughts were something like this:

breakout = 0
while(packetReceived < totalToReceive || breakout != 0):
    data, addr = sock.recvfrom(packetSize)

but then I need to set breakout somehow. Any help would be greatly appreciated.

Answer 1

You can’t share state just by having a global variable in the parent process. This may appear to work, but it only works sometimes; it’s neither reliable nor predictable. Except that on Windows, it reliably and predictably never works; each child will always have its own independent copy of the flag, and therefore you will never quit.

If you really want to do this by sharing a variable, see Sharing state between processes in the docs, but the short version is: You create a multiprocessing.Value. And then you use a multiprocessing.Condition to protect that value against races, because otherwise, there’s no guarantee that the child processes will ever see a change from the parent.

Of course you can fake this by, e.g. creating an mmap of minimum size and just using m[0] as a flag and m.flush() instead of the condition, but that’s not really any simpler.

The alternative way to do this is to use a multiprocessing.Pipe or similar to pass a “shut down now” message. The child processes can each spawn a thread to block on the pipe, or you can toss the pipe and your socket into a select together, or all the other usual tricks.

There may be another, simpler option in this case: don’t use multiprocessing in the first place. Clearly your background task is not CPU-bound, since it’s just looping around reading from a socket, so why not just threading?

Also, it strikes me that you might be able to simplify your design in other ways, which could remove this problem entirely. Do you need a file between the reading and processing jobs instead of, say, a queue, or even just a direct sequential pipeline? Can you toss the user input and the socket into the same event loop (plain old select if user input is stdin and you don’t care about Windows; use a QSocket instead of a socket.socket if user input is a Qt GUI; twisted if you’re willing to learn twisted; etc.). Or, is there real user input, or just “quit now” (or “shut down the socket and process the remaining messages now”), which you could handle with ^C?