[Python-Dev] Module renaming and pickle mechanisms

[Alexandre Vassalotti]

On Sat, May 17, 2008 at 5:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> I'd like to bring a potential problem to attention that is caused
> by the recent module renaming approach:
>
> Object serialization protocols like e.g. pickle usually store the
> complete module path to the object class together with the object.

Thanks for bringing this up. I was aware of the problem myself, but I
hadn't yet worked out a good solution to it.


> It can also happen in storage setups where Python
> objects are stored using e.g. pickle, ZODB being a prominent
> example. As soon as a Python 2.6 application starts writing to
> such storages, Python 2.5 and lower versions will no longer be
> able to read back all the data.
>

The opposite problem exists for Python 3.0, too. Pickle streams
written by Python 2.x applications will not be readable by Python 3.0.
And, one solution to this is to use Python 2.6 to regenerate pickle
stream.

Another solution would be to write a 2to3 pickle converter using the
pickletools module. It is surely not the most elegant or robust
solution, but I could work.

> Now, I think there's a way to solve this puzzle:
>
> Instead of renaming the modules (e.g. Queue -> queue), we leave
> the code in the existing modules and packages and instead add
> the new module names and package structure with pointers and
> redirects to the existing 2.5 modules.

This would certainly work for simple modules, but what about packages?
For packages, you can't use the ``sys.modules[__name__] = Queue`` to
preserve module identity. Therefore, pickle will use the new package
name when writing its streams. So, we are back to the same problem
again.

A possible solution could be writing a compatibility layer for the
Pickler class, which would map new module names to their old at
runtime. Again, this is neither an elegant, nor robust, solution, but
it should work in most cases.

-- Alexandre