Question

I have written a python program to interface with a compiled program (call it ProgramX) that has some idiosyncrasies that are proving difficult to deal with. I need to feed many thousands of input files to ProgramX via my python program. What I would like to do is to grab the output file that ProgramX creates with each run, and rename it something sensible, like inputfilename.output.

The problem comes in the output file that is written by ProgramX — it is named via an unpredictable method, which will write, and “mercilessly overwrite”, the output file if it already exists (which is the case the majority of the time). The saving grace probably comes with the fact that there is a standard prefix to the output files: think ProgramX.notQuiteRandomNumber.

The only think I can think to do is something like this in my bash shell:

PROGRAMXOUTPUT=$(ls -ltr ProgramX* | tail -n -1 | awk '{print $8}')
mv $PROGRAMXOUTPUT input.output

Which does 90% of what I need, but before I program all that bash into a series of Popen statements, is there a better way to do this? This problem feels like something people might have a much better solution than what I’m thinking.

Sidenote: I can grab the program’s standard output without problems, however it’s the output file that I need to grab.

Bonus: I was planning on running a bunch of instantiations of the program in the same directory, so my naive approach above may start to have unforeseen problems. So perhaps something fancy that watches the PID of ProgramX and follows its output.

Answer 1

To do what your shell script above does, assuming you’ve only got one ProgramX* in the current directory:

import glob, os

programxoutput = glob.glob('ProgramX*')[0]
os.rename(programxoutput, 'input.output')

If you need to sort by time, etc., there are ways to do that too (look at os.stat), but using the most recent modification date is a recipe for nasty race conditions if you’ll be running multiple copies of ProgramX concurrently.

I’d suggest instead that you create and change to a new, perhaps temporary directory for each run of ProgramX, so the runs have no possibility of treading on each other. The tempfile module can help with this.