Question

I am trying to create a stream object that triggers a callback function any time data is written to it.

class MonitoredStream():
    def __init__(self, outstream, callback):
        self.outstream = outstream
        self.callback = callback

    def write(self, s):
        self.callback(s)
        self.outstream.write(s)

    def __getattr__(self, attr):
        return getattr(self.outstream, attr)

This works fine when I call the write method directly, but I would love to have it work also when I have a subprocess’ output hooked to the stream. For example:

def f(s):
    print("Write")

p = sub.Popen(["sh", "test.sh"], stdout=MonitoredStream(sys.stdout, f))
p.communicate()

This just sends output directly to sys.stdout, bypassing the write function completely. Is there a way that I can monitor this output also?

Answer 1

I believe the issue here is that subprocess.Popen doesn’t use the Python interface to the pipe – it instead gets the file descriptor and then uses that to write to the pipe directly, which, as you give the attributes of the stdout pipe, means it uses that, bypassing your code.

My best guess at solving this is to make a new in-between pipe that sits in the middle to let you deal with the stream yourself. I would implement this as a context manager:

import sys
import os
from subprocess import Popen
from contextlib import contextmanager

@contextmanager
def monitor(stream, callback):
    read, write = os.pipe()
    yield write
    os.close(write)
    with os.fdopen(read) as f:
        for line in f:
            callback(line)
            stream.write(line)

def f(s):
    print("Write")

with monitor(sys.stdout, f) as stream:
    p = Popen(["ls"], stdout=stream)
    p.communicate()

Although you could, of course, still use a class:

import sys
import os
from subprocess import Popen

class MonitoredStream():
    def __init__(self, stream, callback):
        self.stream = stream
        self.callback = callback
        self._read, self._write = os.pipe()

    def fileno(self):
        return self._write

    def process(self):
        os.close(self._write)
        with os.fdopen(self._read) as f:
            for line in f:
                self.callback(line)
                self.stream.write(line)

def f(s):
    print("Write")

stream = MonitoredStream(sys.stdout, f)
p = Popen(["ls"], stdout=stream)
p.communicate()
print(stream.process())

Although I feel this is less elegant.