Question

My problem is:

I have 3 procs that would like to share config loaded from the same class and a couple of queues. I would like to spawn another proc as a multiprocessing.manager to share those informations.

How can I do that? Could someone purchase a sample code avoiding use of global vars and making use of multiprocessing manager class?

Python docs wasn’t so helpfull 🙁

Answer 1

I found this particular section in the Python multiprocessing docs helpful. The following program:

from multiprocessing import Process, Queue, current_process
import time

def f(q):
    name = current_process().name
    config = q.get()
    print "%s got config: %s" % (name, config)
    print "%s beginning processing at %s" % (name, time.asctime())
    time.sleep(5)
    print "%s completing processing at %s" % (name, time.asctime())

if __name__ == '__main__':
    q = Queue()
    processes = []
    cfg = { 'my' : 'config', 'data' : 'here' }
    for i in range(3):
        p = Process(target=f, args=(q,))
        processes.append(p)
        p.start()
        q.put(cfg)

    for p in processes:
        p.join()

demonstrates the main script (which is the “multiprocessing manager” in your question) creating 3 processes and sending them each a configuration (shown here as a dictionary).

Each process reads the configuration, does its processing (here, just sleep for 5 secs) then terminates. The output from running this script is:

Process-1 got config: {'my': 'config', 'data': 'here'}
Process-1 beginning processing at Tue Jun 23 23:34:23 2009
Process-2 got config: {'my': 'config', 'data': 'here'}
Process-2 beginning processing at Tue Jun 23 23:34:23 2009
Process-3 got config: {'my': 'config', 'data': 'here'}
Process-3 beginning processing at Tue Jun 23 23:34:23 2009
Process-1 completing processing at Tue Jun 23 23:34:28 2009
Process-2 completing processing at Tue Jun 23 23:34:28 2009
Process-3 completing processing at Tue Jun 23 23:34:28 2009