Question

I’m writing a script in Python to ssh into a few computers (about ten) and have them start rendering 3d images from Blender. It works fine except the next computers’s renders won’t start until the previous ones are finished. Is there a way to start the commands and have them all run concurrently on their own machines?

what my code looks like:

import os
path = /home/me
comp1 = ['sneffels','1','2'] #computer name, start frame, end frame
comp2 = ['bierstadt','3','4']
comp3 = ['diente','5','6']

os.system("ssh igp@" + str(comp1[0]) + " blender -b "+ str(path) +" -s " + str(comp1[1]) + " -e " + str(comp1[2]) + " -a")

os.system("ssh igp@" + str(comp2[0]) + " blender -b "+ str(path) +" -s " + str(comp2[1]) + " -e " + str(comp2[2]) + " -a")

os.system("ssh igp@" + str(comp3[0]) + " blender -b "+ str(path) +" -s " + str(comp3[1]) + " -e " + str(comp3[2]) + " -a")

Answer 1

The problem is that os.system doesn’t return until the program is done, and ssh isn’t done until the command you gave it is done.

This is one of many reasons not to use os.system—as the documentation explicitly says:

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes.

In subprocess, you can create a bunch of subprocesses, and then join them all after they’ve all been kicked off. For example:

p1 = subprocess.Popen("ssh igp@" + str(comp1[0]) + " blender -b "+ str(path) +" -s " + str(comp1[1]) + " -e " + str(comp1[2]) + " -a", shell=True)
p2 = subprocess.Popen("ssh igp@" + str(comp2[0]) + " blender -b "+ str(path) +" -s " + str(comp2[1]) + " -e " + str(comp2[2]) + " -a", shell=True)
p3 = subprocess.Popen("ssh igp@" + str(comp3[0]) + " blender -b "+ str(path) +" -s " + str(comp3[1]) + " -e " + str(comp3[2]) + " -a", shell=True)
p1.wait()
p2.wait()
p3.wait()

That probably isn’t the best way to do this. Read the subprocess docs to understand why shell=True and passing a string is usually not as good as passing a list of parameters, other ways to manage your subprocesses, etc.. But meanwhile, this is probably the simplest change from what you already have.

Another alternative is to not shell out to the ssh command in the first place, but instead use something like paramiko to spawn the remote processes from within Python.