Question

I’m not sure why, but yesterday I was testing some multiprocessing code that I wrote and it was working fine. Then today when I checked the code again, it would give me this error:

Exception in thread Thread-5:
Traceback (most recent call last):
  File "C:\Python32\lib hreading.py", line 740, in _bootstrap_inner
    self.run()
  File "C:\Python32\lib hreading.py", line 693, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Python32\lib\multiprocessing\pool.py", line 342, in _handle_tasks
    put(task)
  File "C:\Python32\lib\multiprocessing\pool.py", line 439, in __reduce__
    'pool objects cannot be passed between processes or pickled'
NotImplementedError: pool objects cannot be passed between processes or pickled

The structure of my code goes as follows:
* I have 2 modules, say A.py, and B.py.
* A.py has class defined in it called A.
* B.py similarly has class B.
* In class A I have a multiprocessing pool as one of the attributes.
* The pool is defined in A.__init__(), but used in another method – run()
* In A.run() I set some attributes of some objects of class B (which are collected in a list called objBList), and then I use pool.map(processB, objBList)
* processB() is a module function (in A.py) that receives as the only parameter (an instance of B) and calls B.runInput()
* the error happens at the pool.map() line.

basically in A.py:

class A:
   def __init__(self):
      self.pool = multiprocessing.Pool(7)
   def run(self):
      for b in objBList:
         b.inputs = something
      result = self.pool.map(processB, objBList)
      return list(result)
def processB(objB):
   objB.runInputs()

and in B.py:

class B:
   def runInputs(self):
       do_something()

BTW, I’m forced to use the processB() module function because of the way multiprocessing works on Windows.

Also I would like to point out that the error I am getting – that pool can’t be pickled – shouldn’t be referring to any part of my code, as I’m not trying to send the child processes any Pool objects.

Any ideas?

(PS: I should also mention that in between the two days that I was testing this function the computer restarted unexpectedly – possibly after installing windows updates.)

Answer 1

Perhaps your class B objects contain a reference to your A instance.