Question

In python, is it possible to unpickle objects that were pickled as old-style python classes after being converted to new-style ones? (Ie, objects with different “class signatures”). *

For example, imagine some instances were saved as:

class foo: # old style

and then a few more objects were pickled after the class was changed to inherit from object:

class foo(object): # new style

Objects that are pickle.dump‘ed with one can be pickle.load‘ed by the same style class, but neither one will load both. I think that the strategy used by pickle changes based on the inheritance (the class that inherits from object has a __reduce__ method automatically defined, but the one without inheritance doesn’t). When trying to load a with-inheritance old-style pickle from code without inheritance (old-style definition), I get the same argument error as seen in this SO question; even though the second argument is redundant, it still changes the “class signature” and expected arguments, and prevents loading. To solve this, I’d be happy to write an unpickler, although I’m afraid it may involve two separate sub-classed unpicklers a la the docs… If I knew how to correctly unpickle each one I’d be fine doing that.

A little background on my situation… I’m using a TrialHandler class to save and reload button presses and reaction times for behavioral psychology experiments. We refactored the TrialHandler class to inherit from a more abstract, baseTrialHandler, but in doing so temporarily changed the class signature to inherit from object. However, we couldn’t unpickle older trial_handler files, so it was changed back. I’d like to look at data from the same experiment that was run with both versions of the trial handler, and so want to unpickle both types of saved log file in the same script.

Alternatively, if I can’t write a custom unpickler that will unpickle both objects, is there another way to serialize them? I tried dumping straight to yaml, but it looks like I’ll have to register the class as something that can be yaml’ized, anyway.

A full description of the problem with specific errors is on the PsychoPy mailing list. Any blog posts explaining the details intermediate pickling, or even python inheritance, would be most welcome; the closest I found was a good explanation of why pickling is insecure describing pickling as a “simple stack-based virtual machine”, which is nice, but doesn’t get me enough to figure out even the basics of unpicking myself.

Answer 1

Going through parts:

1. there is no thing called “class signature” in Python, although I
   got the idea from your question
2. Not inheriting from “object” in a Python 2 program should be
   considered an error. Classes that do not inherit from object – also
   called “old style” classes where just kept on the language for
   backwards compatibility, when new style classes where introduced in
   Python 2.2 (circa year 2000/2001) – old style classes are a lot less
   consistent and lack many features that make today’s python such a
   nice language.

With that out of the way: unpickle will try to deserialize objects based on the class qualified name – as in <module>.<class>, as seem on the object’s __class__.__name__ attribute at pickling time. So it is possible to have a naive way that on an unpickle exception (it raises TypeError), swap the available class with the same name, and retry the unpickle operation
Again: your classes should inherit from “object”(more specifically, have a “type” based metaclass)

As for the example I mentioned, juust write something along:

try:
    new_obj = pickle.load(data_stream)
except TypeError: # if you know this was caused due to the new version of "Foo" class:
    current_foo = foomodule.Foo
    foomodule.Foo = foomodule.OldFoo
    new_obj = pickle.load(data_stream)
    foomodule.Foo = current_foo