Question

I’m running into some difficulties getting output from a subprocess stdout pipe. I’m launching some third party code via it, in order to extract log output. Up until a recent update of the third party code, everything worked fine. After the update, python has started blocking indefinitely, and not actually showing any output. I can manually launch the third party app fine and see output.

A basic version of the code I’m using:

import subprocess, time
from threading import Thread

def enqueue_output(out):
    print "Hello from enqueue_output"
    for line in iter(out.readline,''):
        line = line.rstrip("\r\n")
        print "Got %s" % line
    out.close()

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
thread = Thread(target=enqueue_output, args=(proc.stdout,))
thread.daemon = True
thread.start()

time.sleep(30)

This works perfectly if I substitute third_party.exe for this script:

import time, sys

while True:
    print "Test"
    sys.stdout.flush()
    time.sleep(1)

So I’m unclear as to magic needs to be done to get this working with the original command.

These are all variants of the subprocess.Popen line I’ve tried with no success:

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=0)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, shell=True)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_CONSOLE)
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.STARTF_USESTDHANDLES | subprocess.STARTF_USESHOWWINDOW
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, startupinfo=si)

Edit 1:
I can’t actually use .communicate() in this case. The app I’m launching remains running for long periods of time (days to weeks). The only way I could actually test .communicate() would be to kill the app shortly after it launches, which I don’t feel would give me valid results.

Even the non-threaded version of this fails:

import subprocess, time
from threading import Thread

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, stderr=subprocess.PIPE)

print "App started, reading output..."
for line in iter(proc.stdout.readline,''):
    line = line.rstrip("\r\n")
    print "Got: %s" % line

Edit 2:
Thanks to jdi, the following works okay:

import tempfile, time, subprocess

w = "test.txt"
f = open("test.txt","a")
p = subprocess.Popen("third_party.exe", shell=True, stdout=f,
                        stderr=subprocess.STDOUT, bufsize=0)

time.sleep(30)

with open("test.txt", 'r') as r:
    for line in r:
        print line
f.close()

Answer 1

First I would recommend that you simplify this example to make sure you can actually read anything. Remove the complication of the thread from the mix:

proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
print proc.communicate()

If that works, great. Then you are having problems possibly with how you are reading the stdout directly or possibly in your thread.

If this does not work, have you tried piping stderr to stdout as well?

proc = subprocess.Popen("third_party.exe", 
                        stdout=subprocess.PIPE, 
                        stderr=subprocess.STDOUT, bufsize=1)

Update

Since you say communicate() is deadlocking, here is another approach you can try to see if its a problem with the internal buffer of subprocess…

import tempfile
import subprocess

w = tempfile.NamedTemporaryFile()
p = subprocess.Popen('third_party.exe', shell=True, stdout=w, 
                        stderr=subprocess.STDOUT, bufsize=0)

with open(w.name, 'r') as r:
    for line in r:
        print line
w.close()