pickle's backward compatibility

[exarkun at twistedmatrix.com]

On 02:48 pm, mal at egenix.com wrote:
>exarkun at twistedmatrix.com wrote:
>>On 03:17 pm, pengyu.ut at gmail.com wrote:
>>>Hi,
>>>
>>>If I define my own class and use pickle to serialize the objects in
>>>this class, will the serialized object be successfully read in later
>>>version of python.
>>>
>>>What if I serialize (using pickle) an object of a class defined in
>>>python library, will it be successfully read in later version of
>>>python?
>>
>>Sometimes.  Sometimes not.  Python doesn't really offer any guarantees
>>regarding this.
>
>I think this needs to be corrected: the pickle protocol versions are
>compatible between Python releases, however, there are two things to
>consider:
>
>* The default pickle version sometimes changes between minor
>   releases.
>
>   This is easy to handle, though, since you can provide the pickle
>   protocol version as parameter.
>
>* The pickle protocol has changed a bit between 2.x and 3.x.
>
>   This is mostly due to the fact that Python's native string
>   format changed to Unicode in 3.x.

The pickle protocol isn't the only thing that determines whether an 
existing pickle can be loaded.  Consider this very simple example of a 
class which might exist in Python 2.x:

    class Foo:
        def __init__(self):
            self._bar = None

        def bar(self):
            return self._bar

Nothing particularly fancy or interesting going on there.  Say you write 
a pickle that includes an instance of this class.

Now consider this modified version of Foo from Python 2.(x+1):

    class Foo(object): # The class is new-style now, because someone felt 
like
                       # making it new style

        def __init__(self, baz):  # The class requires an argument to 
__init__
                                  # now to specify some new piece of info

             self.barValue = None  # _bar was renamed barValue because 
someone
                                   # thought it would make sense to 
expose the
                                   # info publically

             self._baz = baz

        def bar(self):
             return self.barValue  # Method was updated to use the new 
name of
                                   # the attribute

Three fairly straightforward changes.  Arguably making Foo new style and 
adding a required __init__ argument are not backwards compatible changes 
to the Foo class itself, but these are changes that often happen between 
Python releases.  I think that most people would not bother to argue 
that renaming "_bar" to "barValue" is an incompatibility, though.

But what happens when you try to load your Python 2.x pickle in Python 
2.(x+1)?

First, you get an exception like this:

  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.5/pickle.py", line 1374, in loads
      return Unpickler(file).load()
    File "/usr/lib/python2.5/pickle.py", line 858, in load
      dispatch[key](self)
    File "/usr/lib/python2.5/pickle.py", line 1070, in load_inst
      self._instantiate(klass, self.marker())
    File "/usr/lib/python2.5/pickle.py", line 1060, in _instantiate
      value = klass(*args)
  TypeError: in constructor for Foo: __init__() takes exactly 2 arguments 
(1 given)

But let's say the class didn't get changed to new-style after all... 
Then you can load the pickle, but what happens when you try to call the 
bar method?  You get this exception:

  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 6, in bar
  AttributeError: Foo instance has no attribute 'barValue'

So these are the kinds of things I am talking about when I say that 
there aren't really any guarantees.

Jean-Paul