Question

I have a python script which calls a lot of shell functions. The script can be run interactively from a terminal, in which case I’d like to display output right away, or called by crontab, in which case I’d like to email error output.

I wrote a helper function for calling shell functions:

import subprocess
import shlex
import sys

def shell(cmdline, interactive=True):
    args = shlex.split(cmdline.encode("ascii"))
    proc = subprocess.Popen(args, stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)
    val = proc.communicate()
    if interactive is True:
        if proc.returncode:
            print "returncode " + str(proc.returncode)
            print val[1]
            sys.exit(1)
        else:
            print val[0]
    else:
        if proc.returncode:
            print ""
            # send email with val[0] + val[1]

if __name__ == "__main__":
    # example of command that produces non-zero returncode
    shell("ls -z")

The problem I’m having is two-fold.

1) In interactive mode, when the shell command takes a while to finish (e.g. few minutes), I don’t see anything until the command is completely done since communicate() buffers output. Is there a way to display output as it comes in, and avoid buffering? I also need a way to check the returncode, which is why I’m using communicate().

2) Some shell commands I call can produce a lot of output (e.g. 2MB). The documentation for communicate() says “do not use this method if the data size is large or unlimited.” Does anyone know how large is “large”?

Answer 1

1) When you use communicate, you capture the output of the subprocess so nothing is sent to your standard output. The only reason why you see the output when the subprocess is finished is because you print it yourself.

Since you want to either see it as it runs and not capture it or capture everything and do something with it only at the end, you can change the way it works in interactive mode by leaving stdout and stderr to None. This makes the subprocess use the same streams as your program. You’ll also have to replace the call to communicate with a call to wait:

if interactive is True:
    proc = subprocess.Popen(args)
    proc.wait()
    if proc.returncode:
        print "returncode " + str(proc.returncode)
        sys.exit(1)
else:
    proc = subprocess.Popen(args, stdout=subprocess.PIPE,
        stderr=subprocess.PIPE)
    val = proc.communicate()
    if proc.returncode:
        print ""
        # send email with val[0] + val[1]

2) Too large is “too large to store in memory”, so it all depends on a lot of factors. If storing temporarily 2MB of data in memory is fine in your situation, then there’s nothing to worry about.