Question

How do I write to a parent’s dictionary? I’ve given a list of urls to some children to retrieve, which then need to write to the parent’s dictionary:

from multiprocessing import Pool
import random

parent_dict={}
urls = ['www.yahoo.com','www.google.com','www.microsoft.com','www.apple.com','www.cisco.com']

def workit(url):
    # retrieve the urls, process some stuff and then add that info to parent_dict
    key = random.randrange(1,10) # pretend that this is the variable that we want in parent_dict
    value = random.randrange(11,20)
    parent_dict[key] = value

pool = Pool(processes = 5)
pool.map(workit,urls)


print parent_dict # returns {}}

Answer 1

The following code is more or less yours, adapted with to deal with passing around the manager.dict that is mentioned in the python multiprocessing docs.

from multiprocessing import Pool
from multiprocessing import Manager
import random

manager = Manager()
dproxy = manager.dict()

urls = ['www.yahoo.com','www.google.com','www.microsoft.com','www.apple.com','www.cisco.com']



def f(url):
    shared_dict = dproxy[0]
    key = random.randrange(1,10)
    value = random.randrange(11,20)
    shared_dict[key] = value
    dproxy[0] = shared_dict

if __name__ == '__main__':
    dproxy[0] = {}
    pool = Pool(processes=5)

    pool.map(f, urls)
    print dproxy[0]

There are a couple of things to note here that are critical issues:

1. This does not always create a dictionary with the same length. Thinking what you are probably thinking right now (some keys were overwritten), I changed the dictionary to a list and the shared_dict[key] = value to shared_list.append((key,value)). There is overwriting going on but the program is actually writing a variable number of pairs too. Sometimes it is three, sometimes two, sometimes four. I’ve added the code here for your edification but I would not actually use the code I posted to do anything, ever
2. This doesn’t behave well when run from idle. You’ll want to run it from a command line using python .py instead.
3. Did I mention that you should not use this code aside from testing to say “huh, that did work rather poorly, didn’t it?” If you can find your way around the issue then more power to you but I have to imagine there’s a better way to set this up that sidesteps the issue entirely.

Good luck with whatever you decide. Don’t use my code. It runs and won’t give you malware but that’s about all it’s got in its favor.