Question

Considering the code below. I would like to run 3 experiments at a time. The experiments are independent, the only thing they share is the Model object which they only read.

As there are seemingly no hard things in threading this out, how can I best do this in Python? I would like to use a pool or so to make sure that only three experiments run at a time. Shall I use multi-processing? If yes, how is the shortest and most concise?

#!/usr/bin/env python2.6
import time

class Model:
    name = ""
    def __init__(self,name):
        self.name = name

class Experiment:
    id = 0
    model = None
    done = False

    def __init__(self,id,model):
        self.id = id
        self.model = model

    def run(self):
        for _ in range(0,60):
            print "Hey %s from experiment %d" % (self.model.name, id)
            time.sleep(1)
        self.done = True


if __name__ == "__main__":
    experiments = []
    model = Model("statictistical model")
    for i in range(0,5):
        experiments.append(Experiment(i, model))

    #How to run 3 experiments at the same time

Answer 1

Check the docs, specifically:

http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool

There really are a lot of examples there that should get you on your way. For instance, I could come up with:

#!/usr/bin/env python2.6
import time
import multiprocessing

class Model:
    name = ""
    def __init__(self,name):
        self.name = name

def run_experiment(id, model):
    print "Experiment %d is starting" % id
    for _ in range(0,60):
        print "Hey %s from experiment %d" % (model.name, id)
        time.sleep(1)
    print "Experiment %d is done" % id
    return "Result for %d" % id


if __name__ == "__main__":
    model = Model("statictistical model")
    experiments = ((i, model) for i in range(0, 5))
    pool = multiprocessing.Pool(3)

    results = [pool.apply_async(run_experiment, experiment) for experiment in experiments]
    for result in results:
        r = result.get()
        # do something with r
        # or nothing, i suppose...

Do also pay attention to what the docs say about using the multiprocessing module:

Functionality within this package
requires that the __main__ method be
importable by the children. This is
covered in Programming guidelines
however it is worth pointing out here.
This means that some examples, such as
the multiprocessing.Pool examples
will not work in the interactive
interpreter