Question

Here is my class that I need to pickle:

class sdict(dict):
    def __getattr__(self, attr):
        return self.get(attr, None)
    __setattr__= dict.__setitem__
    __delattr__= dict.__delitem__
    __reduce__ = dict.__reduce__

It seems to me that __reduce__ should take care of pickling, but instead:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/daniyar/work/Apr24/<ipython-input-28-1cc94b920737> in <module>()
----> 1 pickle.dumps(sdict(a=1))

/usr/lib/python2.6/pickle.pyc in dumps(obj, protocol)
   1364 def dumps(obj, protocol=None):
   1365     file = StringIO()
-> 1366     Pickler(file, protocol).dump(obj)
   1367     return file.getvalue()
   1368 

/usr/lib/python2.6/pickle.pyc in dump(self, obj)
    222         if self.proto >= 2:
    223             self.write(PROTO + chr(self.proto))
--> 224         self.save(obj)
    225         self.write(STOP)
    226 

/usr/lib/python2.6/pickle.pyc in save(self, obj)
    304             reduce = getattr(obj, "__reduce_ex__", None)
    305             if reduce:
--> 306                 rv = reduce(self.proto)
    307             else:
    308                 reduce = getattr(obj, "__reduce__", None)

/usr/lib/python2.6/copy_reg.pyc in _reduce_ex(self, proto)
     82             dict = None
     83     else:
---> 84         dict = getstate()
     85     if dict:
     86         return _reconstructor, args, dict

TypeError: 'NoneType' object is not callable

ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (171, 0))

Should I define __getstate__ too? I’m totally lost with all these multiple methods that can be used for custom pickling: __getinitargs__, __getstate__, __reduce__, etc…

Google only refers me to the official pickle documentation that I find very unclear. Can someone clarify that for me or point me to a good pickle tutorial?

Answer 1

I question whether you are headed down a good path.

But, to fix the pickle problem:

class sdict(dict):
    def __getattr__(self, attr):
        if attr.startswith('__'):
            raise AttributeError
        return self.get(attr, None)
    __setattr__= dict.__setitem__
    __delattr__= dict.__delitem__

Pickle is trying to get state from your object, and dict doesn’t implement it. Your __getattr__ is being called with "__getstate__" to find that magic method, and returning None instead of raising AttributeError as it should. If you fix your __getattr__, pickle works properly.

In fact, why are you returning None for missing attributes anyway? Is that really what you want your object to do?

This is an example of the difficulty of faking builtins as you are trying to do. There is subtle stuff going on under the covers, and you will have to be involved with it.

PS: an identical question.