Question

I want to parallelize a function using IPython.parallel, and when I define it
in the IPython shell it works flawlessly:

Type:       function
Base Class: <type 'function'>
String Form:<function gradient at 0x3ae0398>
Namespace:  Interactive
File:       /root/<ipython-input-30-cf7eabdfef84>
Definition: gradient(w) 
Source:
def gradient(w):
    s = (1.0 + exp(y * (X * w)))**-1
    return C*X.T*((1 - s) * y)

rc = Client() 
rc[:].apply_sync(gradient, w)
...

However, when I define it in a module and use import:

Type:       function
Base Class: <type 'function'>
String Form:<function gradient at 0x3933d70>
Namespace:  Interactive
File:       /root/mv.py
Definition: mv.gradient(w)
Source:
def gradient(w):
    s = (1.0 + exp(y * (X * w)))**-1
    return C*X.T*((1 - s) * y)

import mv 
rc = Client()
rc[:].apply_sync(mv.gradient, w)

CompositeError: one or more exceptions from call to method: gradient
[0:apply]: NameError: global name 'y' is not defined
[1:apply]: NameError: global name 'y' is not define

Furthermore, it works fine one my local system running Python 2.7.2/IPython 0.12, while
it crashes on Python 2.7.2+/IPython 0.12 using the newest Starcluster Ubuntu AMI.

What is going on here?

UPDATE: I installed the IPython 0.13.dev version from github and now it works.

Answer 1

The difference is module globals. When a function is defined in a module, the global namespace is that of the module (i.e. mv.y). When that module is __main__, e.g. an interactively defined function, then the global namespace is your user_ns on the Engine, and is affected by execute("y=5").

IPython provides a decorator, if you want to define functions in modules that should behave as if they are interactively defined (have access to the user namespace as globals):

# mymod

from IPython.parallel.util import interactive

@interactive
def by_a(b):
    """multiply a by b"
    return a*b

And interactively, you can do:

from mymod import by_a

e0 = rc[0]
e0.execute("a=5")
print e0.apply_sync(by_a, 10) # 50
e0.execute("a=10")
print e0.apply_sync(by_a, 10) # 100