Question

I have code like the following,

class _Process(multiprocessing.Process):

    STOP = multiprocessing.Manager().Event()

    def __init__(self, queue, process_fn):

        self._q = queue
        self._p = process_fn
        super().__init__()

    def run(self):

        while True:
            dat = self._q.get()
            if not dat is _Process.STOP:
                self._p(dat, self._q)
                self._q.task_done()
            else:
                self._q.task_done()
                break

but, I cannot compare STOP successfully. This isn’t that surprising when I’m using is since, I believe, is compares object id’s and from the docs ” … This is the address of the object in memory.” So, since I’m using multiple processes the memory address will be different. (I can’t compare it with == either though, and I’m not sure why this is).

This happens with any object I create with Manager(), but if I use a “true” singleton (True or False or None) it does work. Although that’s not an appropriate solution since any of those values may be valid in the queue.

So how can I create a variable, like the singletons, that can be compared across processes?

(N.B. I have tried using a dedicated class too, but get errors about it not being able to be pickled.)

Update: The answer does seem to be to use a class, but I was receiving the pickleing problems as I was only trying with an inner class. Moving it to module scope fixed the error and it works fine. – Thanks @Schnouki!

---

Here’s an example (and pointless) usage of the code, that shows the error …

def f(data, queue):
    print(data)

q = multiprocessing.JoinableQueue()

for i in range(4):
    p = _Process(q, f)
    p.daemon = True
    p.start()
    q.put(i)

q.join()

for i in range(4):
    q.put(_Process.STOP)

q.join()

Answer 1

This is a weird way to use an Event object… If you can’t use None or a boolean, I suggest you use a dedicated class and test the type of what you get from the queue:

class StopProcessing(object):
    pass

#...

q.put(StopProcessing())

#...

while True:
    dat = self._q.get()
    if type(dat) is StopProcessing:
        # ...

Or, of course, you could just keep using the multiprocessing.Event and test for its type. However this would probably be quite misleading for someone else reading your code; using a dedicated type seems much cleaner and Pythonic to me.

EDIT: Ok, so apparently this doesn’t work because the new class is not picklable. So here’s another idea: what if you directly put the type inside your queue, like this:

class StopProcessing(object):
    pass
#...
q.put(StopProcessing)
#...
while True:
    dat = self._q.get()
    if dat is StopProcessing:
        #...

According to the pickle doc, “classes that are defined at the top level of a module” can be pickled.