Question

I’m writing software that runs a bunch of different programs (via twisted’s twistd); that is N daemons of various kinds must be started across multiple machines. If I did this manually, I would be running commands like twistd foo_worker, twistd bar_worker and so on on the machines involved.

Basically there will be a list of machines, and the daemon(s) I need them to run. Additionally, I need to shut them all down when the need arises.

If I were to program this from scratch, I would write a “spawner” daemon that would run permanently on each machine in the cluster with the following features accessible through the network for an authenticated administrator client:

• Start a process with a given command line. Return a handle to manage it.
• Kill a process given a handle.
• Optionally, query stuff like cpu time given a handle.

It would be fairly trivial to program the above, but I cannot imagine this is a new problem. Surely there are existing solutions to doing exactly this? I do however lack experience with server administration, and don’t even know what the related terms are.

What existing ways are there to do this on a linux cluster, and what are some of the important terms involved? Python specific solutions are welcome, but not necessary.

Another way to put it: Given a bunch of machines in a lan, how do I programmatically work with them as a cluster?

Answer 1

The most familiar and universal way is just to use ssh. To automate you could use fabric.

To start foo_worker on all hosts:

$ fab all_hosts start:foo_worker

To stop bar_worker on a particular list of hosts:

$ fab -H host1,host2 stop:bar_worker

Here’s an example fabfile.py:

from fabric.api import env, run, hide # pip install fabric

def all_hosts():
    env.hosts = ['host1', 'host2', 'host3']

def start(daemon):
    run("twistd --pid %s.pid %s" % (daemon, daemon))

def stop(daemon):
    run("kill %s" % getpid(daemon))

def getpid(daemon):
    with hide('stdout'):
        return run("cat %s.pid" % daemon)

def ps(daemon):
    """Get process info for the `daemon`."""
    run("ps --pid %s" % getpid(daemon))

There are a number of ways to configure host lists in fabric, with scopes varying from global to per-task, and it’s possible mix and match as needed..

To streamline the process management on a particular host you could write initd scripts for the daemons (and run service daemon_name start/stop/restart) or use supervisord (and run supervisorctl e.g., supervisorctl stop all). To control “what installed where” and to push configuration in a centralized manner something like puppet could be used.