Question

I’m trying to start 6 threads each taking an item from the list files, removing it, then printing the value.

from multiprocessing import Pool

files = ['a','b','c','d','e','f']

def convert(file):
    process_file = files.pop()
    print process_file

if __name__ == '__main__':

    pool = Pool(processes=6)
    pool.map(convert,range(6))

The expected output should be:

a
b
c
d
e
f

Instead, the output is:

f
f
f
f
f
f

What’s going on? Thanks in advance.

Answer 1

Part of the issue is that you are not dealing with the multiprocess nature of pool, (note that in Python, MultiThreading does not gain performance due to Global Interpreter Lock).

Is there a reason why you need to alter the original list? You current code does not use the iterable passed in, and instead edits a shared mutable object, which is DANGEROUS in the world of concurrency. A simple solution is as follows:

from multiprocessing import Pool

files = ['a','b','c','d','e','f']

def convert(aFile):
    print aFile

if __name__ == '__main__':

    pool = Pool() #note the default will use the optimal number of workers
    pool.map(convert,files)

Your question really got me thinking, so I did a little more exploration to understand why Python behaves in this way. It seems that Python is doing some interesting black magic and deepcopying (while maintain the id, which is non-standard) the object into the new process. This can be seen by altering the number or processes used:

from multiprocessing import Pool

files = ['d','e','f','a','b','c',]

a = sorted(files)
def convert(_):
    print a == files
    files.sort()
    #print id(files) #note this is the same for every process, which is interesting

if __name__ == '__main__':

    pool = Pool(processes=1) #
    pool.map(convert,range(6))

==> all but the first invocation print ‘True’ as expected.

If you set the number or processes to 2, it is less deterministic, as it depends on which process actually executes their statement(s) first.