Question

I recently want to develop a HTTP server frontier to wrap my ipcontroller/ipengine clustering program. The server is a simple derived from BaseHTTPServer. When the server receives a HTTP Get request, its do_GET method will call several mec.execute() method to finish the job. Here is the code sample.

do_GET
{
b = parameter
mec.scatter("a", b)
mec.execute("c=fun(a)")
d = mec.gather("c")
write d
}

Will I face Synchronization problem at the statement mec.execute(“c=fun(a)”)? From my guessing, a variable “c” will be created at each ipengie with value “fun(a)”. If two threads call do_Get method simultaneously with different parameter, what will be the value of “c” at each of ipengine.

Answer 1

If you can express the task as single parallel function call, then you should be safe, because no other requests can sneak in between (and engine globals need not be touched), e.g:

from IPython import parallel

rc = parallel.Client()
view = rc[:]

@view.parallel(block=True)
def pfun(a):
    """each engine will get a chunk of a, not the whole thing"""
    c = fun(a)
    return c

# a will be scattered and c will be gathered
c = pfun(a)

But if not, then the easiest solution is probably to ensure that you don’t have name collisions across jobs by giving your variables for a given request a unique suffix with a UUID:

import uuid
suffix = str(uuid.uuid4()).replace('-','') # remove '-' so we have a valid identifier
a_name = "a_" + suffix
c_name = "c_" + suffix
mec.scatter(a_name, b)
mec.execute("%s = fun(%s)" % (c_name, a_name))
d = mec.gather(c_name)