Question

The codes are like this:

from subprocess import Popen, PIPE


p1 = Popen("command1", stdout = PIPE)
p2 = Popen("command2", stdin = p1.stdout, stdout = PIPE)
result_a = p2.communicate()[0]

p1_again = Popen("command1", stdout = PIPE)
p3 = Popen("command3", stdin = p1_again.stdout, stdout = PIPE)
result_b = p3.communicate()[0]

with open("test") as tf:
    p1_again_again = Popen("command1", stdout = tf)
    p1_again_again.communicate()

The bad part is:

The command1 was executed three times because when I use commnnicate once, the stdout of that Popen object can’t be used again. I was just wondering whether there’s a method to reuse the intermediate results of PIPE.

Does anyone have ideas about how to make these codes better (better performance as well as less lines of codes)? Thanks!

Answer 1

here is a working solution. I have put example commands for cmd1, cmd2, cmd3 so that you can run it. It just takes the output from the first command and uppercases it in one command and lowercases it in the other.

code

from subprocess import Popen, PIPE, check_output
from tempfile import TemporaryFile

cmd1 = ['echo', 'Hi']
cmd2 = ['tr', '[:lower:]', '[:upper:]']
cmd3 = ['tr', '[:upper:]', '[:lower:]']

with TemporaryFile() as f:
    p = Popen(cmd1, stdout=f)
    ret_code = p.wait()
    f.flush()
    f.seek(0)
    out2 = Popen(cmd2, stdin=f, stdout=PIPE).stdout.read()
    f.seek(0)
    out3 = Popen(cmd3, stdin=f, stdout=PIPE).stdout.read()
    print out2, out3

output

HI
hi

some of the things to make note of in the solution. the tempfile module is always a great way to go when needing to work with temp files, it will automatically delete the temporary file as a cleanup once the with statement exits, even if there was some io exception thrown through out the with block. cmd1 is run once and output to the temp file, one calls the wait() method to make sure all execution has completed, then we do seek(0) each time so that when we call the read() method on f it is back at the start of the file. As a reference the question Saving stdout from subprocess.Popen to file, helped me in getting the first part of the solution.