Question

I am trying to use rsync with python. I have read that the preferred way to passing arguments to Popen is using an array.

The code I tried:

p = Popen(["rsync",
        "\"{source}\"".format(source=latestPath),
        "\"{user}@{host}:{dir}\"".format(user=user, host=host, dir=dir)],
        stdout=PIPE, stderr=PIPE)

The result is rsync asking for password, even though I have set up SSH keys to do the authentication.

I think this is a problem with the environment the new process gets executed in. What I tried next is:

p = Popen(["rsync", 
   "\"{source}\"".format(source=latestPath),
   "\"{user}@{host}:{dir}\"".format(user=user, host=host, dir=dir)],
   stdout=PIPE, stderr=PIPE, shell=True)

This results in rsync printing the “correct usage”, so the arguments are passed incorrectly to rsync. I am not sure if this is even supposed to work(passing an array with shell=True)

If I remove the array altogether like this:

p = Popen("rsync \"{source}\" \"{user}@{host}:{dir}\"".format(
        source=latestPath, user=user, host=host, dir=dir),
      stdout=PIPE, stderr=PIPE, shell=True)

The program works fine. It really doesn’t matter for the sake of this script, but I’d like to know what’s the difference? Why don’t the other two(mainly the first one) work?

Is it just that the shell environment is required, and the second one is incorrect?

EDIT: Contents of the variables

latestPath='/home/tomcat/.jenkins/jobs/MC 4thworld/workspace/target/FourthWorld-0.1-SNAPSHOT.jar'
user='mc'
host='192.168.0.32'
dir='/mc/test/plugins/'

Answer 1

I’d like to know what’s the difference?

When shell=True, the entire command is passed to the shell. The quotes are there so the shell can correctly pick the command apart again. In particular, passing

foo "bar baz"

to the shell causes it to parse the command as (Python syntax) ['foo', 'bar baz'] so that it can execute the foo command with the argument bar baz.

By contrast, when shell=False, Python will pass the arguments in the list to the program immediately. For example, try the following subprocess commands:

>>> import subprocess
>>> subprocess.call(["echo", '"Hello!"'])
"Hello!"
0
>>> subprocess.call('echo "Hello!"', shell=True)
Hello!
0

and note that in the first, the quotes are echoed back at you by the echo program, while in the second case, the shell has stripped them off prior to executing echo.

In your specific case, rsync gets the quotes but doesn’t know how it’s supposed to handle them; it’s not itself a shell, after all.