Question

I am trying to run an executable and capture its output using subprocess.Popen; however, I don’t seem to be getting all of the output.

import subprocess as s
from subprocess import Popen 
import os

ps = Popen(r'C:\Tools\Dvb_pid_3_0.exe', stdin = s.PIPE,stdout = s.PIPE)
print 'pOpen done..'

while:

line = ps.stdout.readline()
print line

It prints two line less than the original exe file when opened manually.

I tried an alternative approach with the same result:

f = open('myprogram_output.txt','w')
proc = Popen('C:\Tools\Dvb_pid_3_0.exe ', stdout =f)

line = proc.stdout.readline()
print line
f.close()

Can anyone please help me to get the full data of the exe?

As asked by Sebastian:

Original exe file last few lines o/p:

-Gdd : Generic count (1 – 1000)

-Cdd : Cut start at (0 – 99)
-Edd : Cut end at (1 – 100)

Please select the stream file number below:

1 – .\pdsx100-bcm7230-squashfs-sdk0.0.0.38-0.2.6.0-prod.sao.ts

The o/p I get after running:

-P0xYYYY      : Pid been interested                                          

-S0xYYYY : Service ID been interested
-T0xYYYY : Transport ID been interested
-N0xYYYY : Network ID been interested
-R0xYYYY : A old Pid been replaced by this PID
-Gdd : Generic count (1 – 1000)

So we can see some lines missing. I have to write 1 and choose value after please select the fule number below appears.

I tried to use ps.stdin.write(‘1\n’). It didn’t print the value in the exe file

New code:

#!/usr/bin/env python
from subprocess import Popen, PIPE

cmd = r'C:\Tools\Dvb_pid_3_0.exe'
p = Popen(cmd, stdin=PIPE, stdout=None, stderr=None, universal_newlines=True)
stdout_text, stderr_text = p.communicate(input="1\n\n")

print("stdout: %r\nstderr: %r" % (stdout_text, stderr_text))
if p.returncode != 0:
    raise RuntimeError("%r failed, status code %d" % (cmd, p.returncode))

Thanks Sebastien. I am able to see the entire output but not able to feed in any input with the current code.

Answer 1

To get all stdout as a string:

from subprocess import check_output as qx

cmd = r'C:\Tools\Dvb_pid_3_0.exe'
output = qx(cmd)

To get both stdout and stderr as a single string:

from subprocess import STDOUT

output = qx(cmd, stderr=STDOUT)

To get all lines as a list:

lines = output.splitlines()

To get lines as they are being printed by the subprocess:

from subprocess import Popen, PIPE

p = Popen(cmd, stdout=PIPE, bufsize=1)
for line in iter(p.stdout.readline, ''):
    print line,
p.stdout.close()
if p.wait() != 0:
   raise RuntimeError("%r failed, exit status: %d" % (cmd, p.returncode))

Add stderr=STDOUT to the Popen() call to merge stdout/stderr.

Note: if cmd uses block-buffering in the non-interactive mode then lines won’t appear until the buffer flushes. winpexpect module might be able to get the output sooner.

To save the output to a file:

import subprocess

with open('output.txt', 'wb') as f:
    subprocess.check_call(cmd, stdout=f)

# to read line by line
with open('output.txt') as f:
    for line in f:
        print line,

If cmd always requires input even an empty one; set stdin:

import os

with open(os.devnull, 'rb') as DEVNULL:
    output = qx(cmd, stdin=DEVNULL) # use subprocess.DEVNULL on Python 3.3+

You could combine these solutions e.g., to merge stdout/stderr, and to save the output to a file, and to provide an empty input:

import os
from subprocess import STDOUT, check_call as x

with open(os.devnull, 'rb') as DEVNULL, open('output.txt', 'wb') as f:
    x(cmd, stdin=DEVNULL, stdout=f, stderr=STDOUT)

To provide all input as a single string you could use .communicate() method:

#!/usr/bin/env python
from subprocess import Popen, PIPE

cmd = ["python", "test.py"]
p = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines=True)
stdout_text, stderr_text = p.communicate(input="1\n\n")

print("stdout: %r\nstderr: %r" % (stdout_text, stderr_text))
if p.returncode != 0:
    raise RuntimeError("%r failed, status code %d" % (cmd, p.returncode))

where test.py:

print raw_input('abc')[::-1]
raw_input('press enter to exit')

If your interaction with the program is more like a conversation than you might need winpexpect module. Here’s an example from pexpect docs:

# This connects to the openbsd ftp site and
# downloads the recursive directory listing.
from winpexpect import winspawn as spawn

child = spawn ('ftp ftp.openbsd.org')
child.expect ('Name .*: ')
child.sendline ('anonymous')
child.expect ('Password:')
child.sendline ('[e-mail]')
child.expect ('ftp> ')
child.sendline ('cd pub')
child.expect('ftp> ')
child.sendline ('get ls-lR.gz')
child.expect('ftp> ')
child.sendline ('bye')

To send special keys such as F3, F10 on Windows you might need SendKeys module or its pure Python implementation SendKeys-ctypes. Something like:

from SendKeys import SendKeys

SendKeys(r"""
    {LWIN}
    {PAUSE .25}
    r
    C:\Tools\Dvb_pid_3_0.exe{ENTER}
    {PAUSE 1}
    1{ENTER}
    {PAUSE 1}
    2{ENTER}
    {PAUSE 1}
    {F3}
    {PAUSE 1}
    {F10}
""")

It doesn’t capture output.