Question

I have successfully managed to pipe a variable to a command which is not piped further, in this way:

from subprocess import Popen, PIPE

def exec_command(command, some_input):
    proc = Popen(command, stdout=PIPE, stderr=PIPE, stdin=PIPE)
    (stdout, stderr) = proc.communicate(input=some_input)

… but when trying to pipe it to a further piped command (for example piping to tar, and then further piping to split), it does not seem to work:

from subprocess import Popen, PIPE

def exec_piped_command(command1, command2, some_input):
    proc1 = Popen(command1, stdout=PIPE, stdin=PIPE)
    proc2 = Popen(command1, stdin=proc1.stdout, stdout=PIPE)
    (stdout, stderr) = proc2.communicate(input=some_input)[0]

So, how is the correct way to do this second variant? It seems like the problem with the above code is that the input in the “proc2.communicate()” command does not reach the stdin pipe of proc1? (Not sure though … unfortunatly I’m a bit confused about the subprocess syntax …).

Answer 1

One possibility would be to set up the entire command to be executed by a shell (shell=True to among the keyword args of your Popen() call … and only .communicate() with the ends of the whole pipeline (your input going to command1’s stdin, and your stdout/stderr coming from command2’s stdout/stderr).

A more complicated, but fine grained, approach would be to use the os.fork() along with os.pipe() and os.dup2() and possibly some os.fcntl() calls to set up your own subprocesses, with your own custom plumping, and any of the desired blocking/non-blocking characteristics you need on your file descriptors and then, finally, using your own os.exec* functions in each of these.

This latter approach, obviously, would be duplicating quite a bit of the code that’s already in the subprocess module. However, it provides you the option of doing some of these steps differently in ways that are not exposed via the subprocess.Popen class.

The middle road would be to make a sub class that inherits from subprocess.Popen

Of course it may be preferable to perform some of the parts of this pipeline through Python code and modules (such as tar, gzip, and split operations).