Question

When a process exits abnormally or not at all, I still want to be able to gather what output it may have generated up until that point.

The obvious solution to this example code is to kill the child process with an os.kill, but in my real code, the child is hung waiting for NFS and does not respond to a SIGKILL.

#!/usr/bin/python
import subprocess
import os
import time
import signal
import sys
child_script = """
#!/bin/bash
i=0
while [ 1 ]; do
    echo "output line $i"
    i=$(expr $i \+ 1)
    sleep 1
done
"""
childFile = open("/tmp/childProc.sh", 'w')
childFile.write(child_script)
childFile.close()

cmd = ["bash", "/tmp/childProc.sh"]
finish = time.time() + 3
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
while p.poll() is None:
    time.sleep(0.05)
    if finish < time.time():
        print "timed out and killed child, collecting what output exists so far"
        out, err = p.communicate()
        print "got it"
        sys.exit(0)

In this case, the print statement about timing out appears and the python script never exits or progresses. Does anybody know how I can do this differently and still get output from my child processe

Answer 1

Problem is that bash doesn’t answer to CTRL-C when not connected with a terminal.
Switching to SIGHUP or SIGTERM seems to do the trick:

cmd = ["bash", 'childProc.sh']
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, 
                          stderr=subprocess.STDOUT, 
                          close_fds=True)
time.sleep(3)
print 'killing pid', p.pid
os.kill(p.pid, signal.SIGTERM)
print "timed out and killed child, collecting what output exists so far"
out  = p.communicate()[0]
print "got it", out

Outputs:

killing pid 5844
timed out and killed child, collecting what output exists so far
got it output line 0
output line 1
output line 2