Question

My question is regarding the multiprocessing module of Python.
In the simplest form, my question is the strange behaviour of the following code:

import numpy as np
from multiprocessing import Pool

x = np.random.random(100)
y = np.random.random(100)
y2 = y[:]

def I(i):
    y[i] = x[i]

pool = Pool()
pool.map(I,range(100))

After the execution, my hope is that y = x.
However, we get y = y2. (The assignments are not working.)
Why is this happening?
What is the best way to compute f(x[i]) and assign it to y[i]?

Answer 1

The behavior you’re seeing is not so surprising if you think about what is being synchronized between the processes used by Pool to do your work. Only the arguments and return values of the I function are synchronized in your current code, so it makes sense that x and y keep their original values in the calling process.

I suspect your current code is a minimal test case, which is troublesome because there’s not really a meaningful implementation of copying one array to another using Pool.map. Here’s a trivial solution, but I’m not sure it generalizes to whatever your real task is:

import numpy as np
from multiprocessing import Pool

def I(v):
    return v

if __name__ == "__main__":  # this boilerplate is required on on Windows
    x = np.random.random(100)
    y = np.random.random(100)

    pool = Pool()
    y[:] = pool.map(I, x)

    print(x == y) # [True, True, True, ...]

This passes each value of x through to another process (where nothing is done with it) and the result values are passed back and assigned into y (pool.map returns a list). It’s pretty silly.

A slightly more sophisticated approach might copy x over to the worker processes, using the initializer and initargs arguments in the Pool constructor. Here’s an example that does that:

import numpy as np
from multiprocessing import Pool

def I(index):
    return x[index]

def setup(value):
    global x
    x = value

if __name__ == "__main__":
    x = np.random.random(100)
    y = np.random.random(100)

    pool = Pool(initializer=setup, initargs=(x,))
    y[:] = pool.map(I, range(100))

    print(x == y) # [True, True, True, ...]

Note though that x is only copied one way. If I were to modify its value, the changes would not be synchronized between processes.

If your task is something that really does requires synchronized access to both the source and target array, you might try out multiprocessing.Array. I don’t have any direct experience with it, but it should be possible to replace y with a synchronized version of itself. Unfortunately, I suspect the synchronization will slow your program down, so don’t do it unless you really need to!