Question

Assume a python script is callable with:

python myscript.py ./mycommand > myoutput.txt

How do I make sure that myscript.py gets ./mycommand > myoutput.txt in the list of command line arguments rather than the shell piping the output of python myscript.py ./mycommand to the file?

EDIT

Using quotes poses additional problems. Consider:

import subprocess
import sys

argList = sys.argv[1].split()

subprocess.call(argList)

which is then called by:

python myscript.py 'ls -l > ls.out'

results in

ls: cannot access >: No such file or directory
ls: cannot access ls.out: No such file or directory

The motivation is that only the output from the command I am trying to run should be saved in the file. Any other output from myscript.py to stdout shouldn’t go to the file. It would appear that the pipe needs to be set with the stdout option to subprocess.call(). Is there a way to use the list of arguments directly so I don’t have to parse it for things like >, 2>, 2&>, < etc?

Answer 1

A standard bash or tcsh shell will parse the command you give it itself, and interpret certain special characters as direction to the shell process itself (the “>” character in your example). The called program has no control over this; the command line parsing happens before your program is called.

In your example, you could say:

python myscript.py ./mycommand ">" myoutput.txt

And the quoted “>” will tell the shell not to use its special behavior.

The subprocess module has two modes of operation; one with a shell (typically /bin/sh) intermediate process — this is when you give the Popen object a shell=True keyword argument. When you invoke it in this way, it invokes the shell to parse the argument text, and then the shell invokes the command you specified as a subprocess-once-removed from your program.

The other mode of operation, the default, is when shell=False. This creates a subprocess and populates its ARGV array directly without any intermediate shell trying to interpret the characters in your command.

The end result is, if you want to issue a command from a shell that interprets special characters, and include special characters, you have to escape them or quote them. If you’re invoking a subprocess from the subprocess standard library module, you can pass in arbitrary text as the elements of the subprocess-to-be’s ARGV, and not worry about them being interpreted in some special way by an intermediary shell process.