Question

my python script is the following code:

 1 import subprocess
 2 
 3 # initial 'output' to make
 4 r0 = 21
 5 # number of runs to make
 6 rn = 10
 7 
 8 rf = r0+rn
 9 
10 for i in range(r0, rf):
11   #output directory
12   opt_dir = 'output'+str(i)
13   #put it in this output directory
14   popt_dir = './output'+str(i)
15 
16   subprocess.call(['mkdir', opt_dir])
17   subprocess.call(['./exp_fit', 'efit.inp'])
18   subprocess.call(['mv', 'at*', popt_dir])

the intention is this:

i have a program called “exp_fit” which takes an input file “efit.inp”. one call to ./exp_fit efit.inp will create output files called ‘at0_l0_l0’, ‘at0_l1_l-1’, … etc (total 475 files starting with ‘at’).

now, i have been generating data files by running ‘exp_fit’, then creating output directories and moving them into the output directories with the following bash commands:
(for example, with the 20th run of my code)

mkdir output20

mv at* ./output20

so i would think that my script should do the same thing. however, it only does the following:

(1) it correctly generates all output files (475 files starting with ‘at’)
(2) it correctly creates the desired directories (output21 – output30)
(3) it DOES NOT, however, correctly move all the output files starting with ‘at’ into the desired directories. why is this? shouldn’t the call to line 18 correctly execute the command to move all my files starting with ‘at’ into the desired directory?

should i be writing this script with bash instead of python? what is wrong with this?

Answer 1

The problem is that this subprocess command

subprocess.call(['mv', 'at*', './output20'])

is not the same as typing this at a prompt

$ mv at* ./output20

In the latter case, the bash glob expansion converts the single at* argument to a list of arguments of matching filenames for the mv command. So the kernel sees the second as

['mv', 'at0_l0_l0', 'at0_l1_l-1', './output20']

kev’s answer tells Python to pass the command through the shell, so the escaping will occur.

But the better solution is to use the glob module and os.rename libraries and not call the subprocess. Creating subprocesses is expensive, and using shell=True could lead to security holes, so it’s best to avoid that habit.

(Actually, I suggest making the output directory, switching into it, and then running the exp_fit program from within that directory. Then you won’t have to move the output. Try that first.)