Question

My code seems to hang on this popen call

command = "ls"
commandList = shlex.split(command)
print("Executing " + command +"\n")
print(commandList)
output = "#" * 10 + "\n" + server.name + "\n\n"
process = subprocess.Popen(
    command,
    shell=True,
    stdin=subprocess.PIPE, 
    stdout=subprocess.PIPE)
output = process.communicate()[0]
#scriptRunner.threadComplete(output)
return output

The code is in the executeScriptOverSSH method on the SshWorker

class WorkerThread(threading.Thread):
    def __init__(self, pathToScript, server, runner):
        super(WorkerThread, self).__init__()
        self.scriptRunner = runner
        self.server = server
        self.pathToScript = pathToScript
        self.sshWorker = SshWorker.SshWorker()

    def run(self):
        print('Thread Starting')
        output = self.sshWorker.executeScriptOverSSH(
            self.server, 
            self.pathToScript)
        print('Thread Finishing!')
        self.scriptRunner.threadComplete(output)`

The threads never get past the call to Popen — I’ve checked by using print statements there. Any ideas?

Answer 1

I’ve run the below code and it runs correctly. You mentioned that “check_output barfs if the return code is non zero”, check out the check_output docs it raises a CalledProcessError exception which you can catch and contains all the error details in a nice structure.

import threading, subprocess, random, time, pprint
from subprocess import check_output

class WorkerThread(threading.Thread):
    def __init__(self, user, host, script, runner):
        super(WorkerThread, self).__init__()
        self.user = user
        self.host = host
        self.script = script
        self.runner = runner

    def run(self):
        cmd = "ssh {user}@{host} 'bash -s' < {script}".format(**self.__dict__)
        self.runner.complete(check_output(cmd, shell=True))

class Runner(object):
    def complete(self, output):
        print 'output:\n', output

WorkerThread('marwan', 'homebox', 'local_script.sh', Runner()).start()

The code above follows the same structure you posted in your question. But I don’t recommend you take this approach here are some suggestions:

• From the WorkerThread you call runner, I don’t know the details of runner but when doing multi-threaded programming the most robust thread-safe option is to communicate using a Queue.
• There are a number of excellent libraries that do what you are doing here hand. They are well tested and do it cleanly. paramiko is excellent and Fabric is really great too.

simple fabric remote run example

from fabric.api import run

def anonymous():
    run("uname -a")