Question

I’m using subprocess.popen with shlex to call a remote bash script using ssh. This command works quite fine on bash itself. But as soon as I try to translate it to python and shlex with subprocess.popen it errs out.

Remote bash script:

#!/bin/bash
tmp="";     
while read -r line;
do
    tmp="$tmp $line\n";
done;
echo $tmp;

BASH CMD RESULT(Invoking the remote bash script on the command line)

$> ssh x.x.x.x cat < /tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt " | /path/to/bash/script.sh;"
Bar\n
$> 

Python code

import shlex
import subprocess

fn = '/tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt'
s = """                                                                    
ssh x.x.x.x cat < {localfile} '| /path/to/bash/script.sh;'
""".format(localfile=fn)

print s

lexer = shlex.shlex(s)                                                     
lexer.quotes = "'"                                                         
lexer.whitespace_split = True                                              
sbash = list(lexer)                                                        
print sbash                                                                

# print buildCmd                                                           
proc=subprocess.Popen(sbash,stdout=subprocess.PIPE,stderr=subprocess.PIPE) 
out,err=proc.communicate()                                                 

print "Out: " + out                                                        
print "Err: " + err                                                        

PYTHON SCRIPT RESULT:

$> python rt.py

    ssh x.x.x.x cat < /tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt '| /path/to/bash/script.sh'
['ssh', 'x.x.x.x', 'cat', '<', '/tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt', "'| /path/to/bash/script.sh'"]
Out: 
Err: bash: /tmp/bef69a1d-e580-5780-8963-6a9b950e529f.txt: No such file or directory
$>

What am I missing?

Answer 1

The problem is that you’re using shell redirection in the command, but there’s no shell spawned when using subprocess.

Consider the following (very simple) program:

import sys
print sys.argv

Now if we run it like you’re running ssh (assuming foofile.txt exists), we get:

python argcheck.py ssh cat < foofile.txt " | /path/to/bash/script.sh;"
['argcheck.py', 'ssh', 'cat', ' | /path/to/bash/script.sh;']

Notice that < foofile.txt never make it to python’s commandline arguments. That’s because the bash parser intercepts the < and the file that comes after it and redirects the contents of that file to your program’s stdin. In other words, ssh is reading the file from stdin. You want your file to be passed to stdin of ssh using python as well.

s = """                                                                    
ssh x.x.x.x cat '| /path/to/bash/script.sh;'
"""

#<snip>

proc=subprocess.Popen(sbash,stdout=subprocess.PIPE,stderr=subprocess.PIPE,
                            stdin=subprocess.PIPE)
out,err=proc.communicate(open(fn).read())

will work presumably.

---

The following works for me:

import subprocess
from subprocess import PIPE

with open('foo.h') as f:
    p = subprocess.Popen(['ssh','mgilson@XXXXX','cat','| cat'],stdin=f,stdout=PIPE,stderr=PIPE)
    out,err = p.communicate()
    print out
    print '#'*80
    print err

And the equivalent command in bash:

ssh mgilson@XXXXX cat < foo.h '| cat'

where foo.h is a file on my local machine.