Question

I’m trying to increment a number sequentially in a function that is called by multiprocessing.pool.map(). When I run the following code I get the number incremented the same number of times as their are pools for each number.

import time
import multiprocessing
import decimal
import random

lists = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h','i', 'j', 'k']
def thefunction(listi):
    global number
    number += 1
    time.sleep(decimal.Decimal(random.random()))
    print time.strftime('%H:%M:%S'), number, listi

number = 0
pool = multiprocessing.Pool(4)
pool.map(thefunction, lists)
print number

The results print out like this

01:01:28 1 b
01:01:28 2 e
01:01:28 1 a
01:01:28 1 c
01:01:28 1 d
01:01:28 2 h
01:01:29 2 i
01:01:29 2 g
01:01:29 3 f
01:01:29 3 j
01:01:29 3 k
0

How can I increment the number correctly?

(time.sleep(decimal.Decimal(random.random())) was only added to stop the script printing to the same line)

Answer 1

The reason why the example is not working is that several instances of the counter are being created and incremented separately.

You need to create a shared counter and lock that are initialized appropriately for each process that is started:

import time
from multiprocessing import Pool, Value, Lock
import decimal
import random

number = Value('i', 0)
lock = Lock()
lists = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h','i', 'j', 'k']

def thefunction(listi):
    time.sleep(decimal.Decimal(random.random()))
    with lock:
        number.value += 1
        print time.strftime('%H:%M:%S'), number.value, listi

def initializer(*args):
    global number, lock
    number, lock = args

pool = Pool(4, initializer, (number, lock))
pool.map(thefunction, lists)
print number.value