Question

How can I write to stdout from Python and feed it simultaneously (via a Unix pipe) to another program? For example if you have

# write file line by line
with open("myfile") as f:
  for line in f:
    print line.strip()

But you’d like that to go line by line to another program, e.g. | wc -l so that it outputs the lines in myfile. How can that be done? thanks.

Answer 1

If you want to pipe python to wc externally, that’s easy, and will just work:

python myscript.py | wc -l

If you want to tee it so its output both gets printed and gets piped to wc, try man tee or, better, your shell’s built-in fancy redirection features.

If you’re looking to run wc -l from within the script, and send your output to both stdout and it, you can do that too.

First, use subprocess.Popen to start wc -l:

wc = subprocess.Popen(['wc', '-l'], stdin=subprocess.PIPE)

Now, you can just do this:

# write file line by line
with open("myfile") as f:
  for line in f:
    stripped = line.strip()
    wc.stdin.write(stripped + '\n')

That will have wc‘s output go to the same place as your script’s. If that’s not what you want, you can also make its stdout a PIPE. In that case, you want to use communicate instead of trying to get all the fiddly details right manually.