Question

After spawning a subprocess using the Python subprocess library, I’m using stderr to pass a message from the child process to the parent process containing some serialized data. I then want the parent to return (via stdin) the result of a function applied to this data.

In essence, I have a function inside the subprocess that does something like this:

sys.stderr.write("some stuff to write")
# some time later
some_var = sys.stdin.read()

However, this completes locks the parent while waiting for the stderr input, so I tried to call:

sys.stderr.flush()
os.fsync(sys.stderr.fileno())

However, this doesn’t work. Nothing after the os.fsync is executed. In addition, when I call proc.poll() in the parent process, it appears, the child’s return code is 1.

What can I do to prevent this? Should I consider another approach?

Answer 1

I would consider another approach.
You may use an indipendent process (multiprocessing.Process) and using two queues to communicate with it (multiprocessing.Queue) one for the input and the other one for the output.
Example on starting the process:

import multiprocessing

def processWorker(input, result):
    work = input.get()
    print work
    result.put(work*work)

input  = multiprocessing.Queue()
result = multiprocessing.Queue()

p = multiprocessing.Process(target = processWorker, args = (input, result))
p.start()

input.put(2)
res = result.get(block = True)
print res

Then you may iterate passing again it. Usage of multiprocessing.Queue is more robust since you do not need to rely on stdout/err parsing and you also avoid related limitation.
Moreover you can easily manage more subprocesses.

Then, you can also set a timeout on how long you want a get call to wait at max, eg:

import queue
try:
    res = result.get(block = True, timeout = 10)
except Queue.Empty:
    print error