Question

I’m running Python 3.3 on Mac OS 10.6.8. I am writing a script that runs several subprocesses, and I want to capture the output of each one and record it in a file. I’m having trouble with this.

I first tried the following:

import subprocess
logFile = open("log.txt", 'w')
proc = subprocess.Popen(args, stdout=logFile, stderr=logFile)
proc.wait()

This produced an empty log.txt. After poking around on the internet for a bit, I tried this instead

import subprocess
proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, err = proc.communicate()
logFile = open("log.txt", 'w')
logFile.write(output)

This, too, produced an empty log.txt. So instead of writing to the file, I tried to just print the output to the command line:

output, err = proc.communicate()
print(output)
print(err)

That produced this:

b''
b''

The process I’m trying to run is fastq_quality_trimmer. It takes an input file, filters it, and saves the result to a new file. It only writes a few lines to stdout, like so

Minimum Quality Threshold: 20
Minimum Length: 20
Input: 750000 reads.
Output: 750000 reads.
discarded 0 (0%) too-short reads.

If I run it from the command line and redirect the output like this

fastq_quality_trimmer -Q 33 -v -t 50 -l 20 -i in.fq -o in_trimmed.fq > log.txt

the output is successfully written to log.txt.

I thought perhaps that fastq_quality_trimmer was somehow failing to run when I called it with Popen, but my script produces a filtered file that is identical to the one produced when I run fastq_quality_trimmer from the command line. So it’s working; I just can’t capture the output. To make matters more confusing, I can successfully capture the output of other processes (echo, other Python scripts) using code that is essentially identical to what I’ve posted.

Any thoughts? Am I missing something blindingly obvious?

Answer 1

You forgot a comma:

["fastq_quality_trimmer", "-Q", "33" "-v", "-t", "50", "-l", "20", "-i", leftInitial, "-o", leftTrimmed]

add it between "33" and "-v".

You are essentially passing in the arguments -Q 33-v instead of -Q 33 -v.

Python will concatenate two adjacent strings if there is only whitespace between them:

>>> "33", "-v"
('33', '-v')
>>> "33" "-v"
'33-v'

Since -v is the verbose switch that is required to make fastq_quality_trimmer produce output at all, it’ll remain silent with it missing.

Whenever you encounter problems with calling a subprocess, triple check the command line created. Pre-pending args with ['echo'] can help in that:

proc = subprocess.Popen(['echo'] + args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, err = proc.communicate()
print(output)