Question

I am using python to run some shell scripts, RScripts, python programs etc. These programs may run for a long time and may output a lot of (logging)info to stdout and stderr. I am using the following (Python 2.6) code which works fine:

stdoutFile=open('stdout.txt', 'a')
stderrFile=open('stderr.txt', 'a')
subprocess.call(SHELL_COMMAND, shell=True, stdout=stdoutFile, stderr=stderrFile)
stdoutFile.close()
stderrFile.close()

This is mainly logging info that goes to the files and this info can be generated over long periods of time. Therefore i was wondering if it is possible to prefix each line with date and time?

For example if i would currently log:

Started
Part A done
Part B done
Finished

Then i would like it to be:

[2012-12-18 10:44:23] Started
[2012-12-18 12:26:23] Part A done
[2012-12-18 14:01:56] Part B done
[2012-12-18 22:59:01] Finished

Note: Modifying the programs that i run is not and option as this python code kinda like a wrapper to these programs.

Answer 1

Instead of providing files to the stdout and stderr arguments of subprocess.call(), create a Popen object directly and create PIPEs, then read off those pipes in this manager script and prepend whatever tags you want before writing to whatever log file you want.

def flush_streams_to_logs(proc, stdout_log, stderr_log):
    pipe_data = proc.communicate()
    for data, log in zip(pipe_data, (stdout_log, stderr_log)):
        # Add whatever extra text you want on each logged message here
        log.write(str(data) + '\n')

with open('stdout.txt', 'a') as stdout_log, open('stderr.txt', 'a') as stderr_log:
    proc = subprocess.Popen(SHELL_COMMAND, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    while proc.returncode is None:
        flush_streams_to_logs(proc, stdout_log, stderr_log)
    flush_streams_to_logs(proc, stdout_log, stderr_log)

Note that communicate() blocks until the subprocess exits. You may want to use the subprocess’ streams directly so you have more real-time logging, but then you have to handle concurrency and buffer fill states yourself.