Question

I am trying to use IPython.parallel’s support for SSH to allow my Linux client to run remote ipengines from an OS X server that has EPD64 installed. This fails, however, as it attempts to use my local machine to figure out the right command to run on the remote host, which has its ipengineapp package in a different location. How do I modify ipcluster_config.py to recognize the difference?

Concretely, when I run ipcluster start --log-level=DEBUG on the remote host, I get console output telling me that contains a line like the following:

[IPClusterStart] Starting LocalEngineLauncher: ['/Library/Frameworks/EPD64.framework/Versions/7.3/bin/python', '-c', 'from IPython.parallel.apps.ipengineapp import launch_new_instance; launch_new_instance()', '--profile-dir', u'/Users/username/.ipython/profile_default', '--cluster-id', u'', '--log-to-file', '--log-level=20']

On the other hand, when run from my local machine with ipcluster start --log-level=DEBUG, I get the following line, as would be appropriate for a Linux host:

[IPClusterStart] Starting SSHEngineLauncher: ['ssh', '-tt', u'hostname', '/usr/bin/python', u'/usr/lib/python2.7/site-packages/IPython/parallel/apps/ipengineapp.py', '--profile-dir', u'/home/username/.ipython/profile_ssh', '--log-to-file', '--log-level=20']

My ipcluster_config.py for this example is:

c = get_config()
c.IPClusterEngines.engine_launcher_class = 'SSHEngineSetLauncher'
c.IPClusterStart.controller_launcher_class = 'SSHControllerLauncher'

c.SSHEngineSetLauncher.engines = {
    'hostname1': 12,
    'hostname2': 12,
}

Answer 1

I think this is improved in 0.14, but the config value you are looking for is SSHEngineSetLauncher.engine_cmd.
Edit this in ipcluster_config.py, so it’s something like:

c.SSHEngineSetLauncher.engine_cmd = ['/path/to/your/python', '-c', 'from IPython.parallel.apps.ipengineapp import launch_new_instance; launch_new_instance()']

or, sometimes even the simplest thing works, as long as your PATH is configured in a simple ssh session:

c.SSHEngineSetLauncher.engine_cmd = ['ipengine']