Question

I must be overlooking something terribly obvious. I need to execute a C program, display its output in real time and finally parse its last line, which should be straightforward as the last line printed is always the same.

process = subprocess.Popen(args, shell = True,  
                           stdout = subprocess.PIPE, stderr = subprocess.PIPE)

# None indicates that the process hasn't terminated yet.
while process.poll() is None:

    # Always save the last non-emtpy line that was output by the child
    # process, as it will write an empty line when closing its stdout.
    out = process.stdout.readline()
    if out:
        last_non_empty_line = out

    if verbose:
        sys.stdout.write(out)   
        sys.stdout.flush()

# Parse 'out' here...

Once in a while, however, the last line is not printed. The default value for Popens’s bufsize is 0, so it is supposed to be unbuffered. I have also tried, to no avail, adding fflush(stdout) to the C code just before exiting, but it seems that there is absolutely no need to flush a stream before exiting a program.

Ideas anyone?

Answer 1

readline() has to buffer the text, waiting for a new-line.

You’ll always have a race condition – no amount of un-buffered streams will handle the fact that you’re handling a line, then checking for exit, then reading a line, so if the subprocess exits while you’re handling a line, you won’t read anything else. In addition the shell is probably introducing its own buffering.

So you could either:

1. Use communicate() and give up the verbose output while the subprocess is running.

2. Make sure you continue reading after the process has exited, until you get EOF.

I would also suggest altering your code so that you don’t have to use shell=True.