Question

I am currently using python multiprocess to do some simple parallel programming.
I use an async decorator

def async(decorated):
    module = getmodule(decorated)
    decorated.__name__ += '_original'
    setattr(module, decorated.__name__, decorated)
    def send(*args, **opts):
        return async.pool.apply_async(decorated, args, opts)
    return send

and then

@async
def evalfunc(uid, start, end):
        veckernel(Posx, Posy, Posz, Quant, Delta)
        return (uid, GridVal)

def runit(outdir):
    async.pool = Pool(8)
    results = []
    for uid in range(8):
        result = evalfunc(uid,Chunks[uid], Chunks[uid+1])
            results.append(result)

If I run this on a 8 processor or 8 cores machine it essentially uses only two cores. Why is that? Is there a way to do proper core pinning like with pthreads?

Thanks a lot,
Mark

Answer 1

If the function being called by apply_async (e.g. evalfunc) finishes very quickly, then all worker processes in the pool may not be utilized.

If that is indeed your situation, then you need to pass more data to each call to evalfunc so each process has more to chew on.