Question

I have a simple script with user defined function run that I want to run a couple of times in parallel using Pool.map_async. When I try it I get the following error:

def getKey(where, key):
    found = re.search('<td.*?>%s:</td>.*?<td>(.*?)</td>' % key, where, re.DOTALL).group(1)
    return re.sub('<[^>]*?>', "", found).strip()


def extract(sitename):
    text = urllib.urlopen(sitename).read()
    return getKey(text, 'Name')


def run(start, span):
    links = get(start, span)
    if (len(links) == 0): return

    pool = Pool(span)
    pool.map_async(extract, links).get()
    pool.close()
    pool.join()

    run(start + span, span)

run(0, 50)

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed 

On http://docs.python.org/library/multiprocessing.html it’s stated that Functionality within this package requires that the __main__ module be importable by the children but I don’t understand what that actually means and what should I do to solve this issue. Please advice.

Answer 1

Just add one line in your script before calling run():

if __name__=="__main__":
    run(0, 50)