Question

I’m building a test app on AWS. I have a ‘master’ machine that owns the application. I’d like, upon occasion, to be able to offload processing to some on-demand slave machines. Assuming an instantiated slave, with the bare minimum of required software (not including my main app), how can I ‘send’ functional units of Python code to one or more slaves for execution?

I know and accept that this architecture may not be optimal for many reasons – and I know it would be simpler if I could preinstall the relevant code on my instances at start-up. But let’s assume we have a hard constraint that the slave instances cannot be preloaded beyond the OS and some basic libraries – for example, what if I wanted to squirt some generated code at another machine?

EDIT: StackOverflow is such a SHIT HOT resource! Thanks to everyone for their replies. Awesome. Give me a couple of days and I’ll report back.

Answer 1

I’ve run into this problem several times in various flavors.

Option 1 – eval()

When I want to just hack something out quickly, I use eval() or one of it’s cousins in the stdlib. Transfer the source to your master, then compile and eval:

src = getSourceFromMaster()
obj = compile(src, "master.py", "exec")
exec(obj)

As long as your transport to get source from client to server is trustworthy and the actions the source needs to take are relatively straightforward, this works. A few times I’ve needed tighter integration between the master and slave machines, with lots of back and forth processing or complex data structures. In those cases, I use Pyro.

Option 2 – Pyro

Pyro is a full on, cross-platform remote method execution library for Python. I’ve used it in production environment to send processing from a linux machine to a windows machine and back, and it’s been super stable.

Example from their docs:

Master:

# save this as greeting.py
class GreetingMaker(object):
    def get_fortune(self, name):
        return "Hello, {0}. Here is your fortune message:\n" \
               "Behold the warranty -- the bold print giveth and the fine print taketh away.".format(name)

Slave:

# save this as client.py
import greeting
name=raw_input("What is your name? ")
greeting_maker=greeting.GreetingMaker()
print greeting_maker.get_fortune(name)

output:

$ python client.py
What is your name? Irmen
Hello, Irmen. Here is your fortune message:
Behold the warranty -- the bold print giveth and the fine print taketh away.

The super-awesome thing about Pyro is the “import greeting” line in client.py — that code comes from the server.

If you’re starting from a bare OS install, you can push down a python script to host the client code for either with SSH immediately after you spin up the new instance. Then you’ll have a nice infrastructure to work within.

I cannot speak in anymore detail to the application of either of these to AWS, nor how they compare to the utilities provided by the AWS infrastructure. Welcome ideas or discussion on it.