pickle and module versioning

[Peter Otten]

Neal Becker wrote:

> I find pickle really handy for saving results from my (simulation)
> experiments.  But recently I realized there is an issue.  Reading the
> saved results requires loading the pickle, which in turn will load any
> referenced
> modules.  Problem is, what if the modules have changed?
> 
> For example, I just re-implemented a python module in C++, in a not quite
> compatible way.  AFAIK, my only choice to not break my setup is to choose
> a different name for the new module.
> 
> Has anyone else run into this issue and have any ideas?  I can imagine
> perhaps some kind of module versioning could be used (although haven't
> really thought through the details).

I just tried the renaming part (replacing one.Foo with two.Bar):

$ cat upgrade.py
import pickle
import one

with open("data.dat", "wb") as f:
    pickle.dump(one.Foo(1, 2), f)

with open("data.dat", "rb") as f:
    print(pickle.load(f))

class MyUnpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if (module, name) == ("one", "Foo"):
            module, name = ("two", "Bar")
        return super().find_class(module, name)

with open("data.dat", "rb") as f:
    print(MyUnpickler(f).load())

$ python3.7 upgrade.py
Foo(alpha=1, beta=2)
Bar(alpha=1, beta=2, gamma=42)

In simple cases Bar.__setstate__() should be able to deal with changes in 
object layout. I used

    def __setstate__(self, namespace):
        namespace.setdefault("gamma", 42)
        vars(self).update(namespace)