Question

I am trying to create a dummy wrapper class ( a mongoengine Document ) that implements the interface of Redis hashes. For example:

class HashModel(mongoengine.Document):
    '''
    Represents a dictionary with a name.
    Interface similar to Redis Hashes
    '''
    name = mongoengine.StringField()
    adict = mongoengine.DictField()

    def safe_reload(self): # because it fails if no object present
        try: self.reload()
        except:pass

    def fix_key(self, key): # for mongoengine validation ( if you really read my code, please also answer why does mongoengine need that ) 
        return key.replace(".","").replace("$","") 

    def hset(self, key, value):
        self.safe_reload()
        self.adict["%s" % self.fix_key(key)] = value  
        self.save(safe=True)
        return True

    def hexists(self, key):
        self.safe_reload()
        key = "%s" % self.fix_key(key)
        return key in self.adict 

Then, I use celery to execute some Tasks. Before that, I initialize a HashModel object on which the tasks perform some operations. But because of multiprocessing, I noticed some inconsistencies. Different processes “get” a different “snapshot” of the object, which is natural anyway. To bypass this issue, I re-initialize the object every time, so as to get an “almost” fresh snapshot every time.

Question: Is there a way to avoid re-initialization? Can I add some code, and what code, to my class above, to do that automatically?

EDIT:
Answer: It seems, that the mongoengine.Document.reload() functions does this. I updated my code to show everything that is relevant to my next problem:

Back in the celery tasks, this problem emerged: When I hset(a_key, a_value), then later when I try to check if it exists, sometimes hexists(a_key) returns False. Even if I coerce it to hset, with:

    while True:
        self.handler.hset(a_key, a_value)
        if self.handler.hexists(a_key): 
            break

later at some point (without deleting the a_key elsewhere),sometimes hexists(a_key) still yields False. How come that????

Answer 1

This is a pure race condition issue. My celery workers may simultaneously obtain an instance and perform different operations on it, then save it, no consistency anymore..