Question

I’m having problems redirecting stdio of another program using subprocess module. Just reading from stdout results in hanging, and Popen.communicate() works but it closes pipes after reading/writing. What’s the easiest way to implement this?

I was playing around with this on windows:

import subprocess proc = subprocess.Popen('python -c 'while True: print \'Hi %s!\' % raw_input()'',                         shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE,                         stderr=subprocess.STDOUT) while True:     proc.stdin.write('world\n')     proc_read = proc.stdout.readline()     if proc_read:         print proc_read

Answer 1

Doesn’t fit 100% to your example but helps to understand the underlying issue: Process P starts child C. Child C writes something to its stdout. stdout of C is a pipe which has a 4096 character buffer and the output is shorter than that. Now, C waits for some input. For C, everything is fine.

P waits for the output which will never come because the OS sees no reason to flush the output buffer of C (with so little data in it). Since P never gets the output of C, it will never write anything to C, so C hangs waiting for the input from P.

Fix: Use flush after every write to a pipe forcing the OS to send the data now.

In your case, adding proc.stdin.flush() in the main while loop and a sys.stdout.flush() in the child loop after the print should fix your problem.

You should also consider moving the code which reads from the other process into a thread. The idea here is that you can never know when the data will arrive and using a thread helps you to understand these issues while you write the code which processes the results.

At this place, I wanted to show you the new Python 2.6 documentation but it doesn’t explain the flush issue, either 🙁 Oh well …