Question

I’m trying to pass a custom Queue between processes using the multiprocessing module. The problem is I’m not getting all the methods coming through to the other side. According to the docs.

A proxy object has methods which invoke corresponding methods of its referent (although not every method of the referent will necessarily be available through the proxy). A proxy can usually be used in most of the same ways that its referent can

But it doesn’t say why or a way around it. I’m wondering if anyone knows a way of doing this.

Here’s a little example of what I’m trying to do.

Server:

from multiprocessing.managers import BaseManager
from Queue import Queue


class KeyHandler(Queue):

    def __init__(self, elements=[]):
        Queue.__init__(self)

        for element in elements:
            self.put(element)

    def __iter__(self):
        return iter(self.get, object())

    def __call__(self):
        return self


class QueueManager(BaseManager):
    pass

keyhandler = KeyHandler(range(10))


QueueManager.register('keyhandler', callable=keyhandler)
manager = QueueManager(address=('', 50000), authkey='foobar')
server  = manager.get_server()
server.serve_forever()

Client:

from multiprocessing.managers import BaseManager


class QueueManager(BaseManager):
    pass


QueueManager.register('keyhandler')
manager = QueueManager(address=('', 50000), authkey='foobar')
manager.connect()

keyhandler = manager.keyhandler()

for elem in keyhandler:
    print elem

Traceback:

Traceback (most recent call last):
  File "client2.py", line 14, in <module>
    for elem in keyhandler:
TypeError: 'AutoProxy[keyhandler]' object is not iterable

The __call__ method works but the __iter__ method doesn’t. Can I force/workaround this somehow?

Answer 1

Use proxy. On client add this class declaration(along with importing BaseProxy):

class KeyHandlerProxy(BaseProxy):
    _exposed_ = ('next', 'get')

    def __iter__(self):
        return self

    def next(self):
        o = self._callmethod('get')
        if object() == o:
            raise StopIteration
        return o

And change register(client’s) to this:

QueueManager.register('keyhandler', proxytype=KeyHandlerProxy)