[Python-Dev] Module renaming and pickle mechanisms

[Brett Cannon]

On Mon, May 19, 2008 at 5:08 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2008-05-18 22:24, Brett Cannon wrote:
>>
>> On Sun, May 18, 2008 at 6:14 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> M.-A. Lemburg wrote:
>>>>
>>>> Perhaps I have a misunderstanding of the reasoning behind
>>>> doing the renaming in the 2.x branch, but it appears that
>>>> the only reason is to get used to the new names. That's a
>>>> rather low priority argument in comparison to the breakage
>>>> the renaming will cause in the 2.x branch.
>>>
>>> I think this is the key point here. The possibility of breaking pickling
>>> compatibility never came up during the PEP 3108 discussions, so wasn't
>>> taken
>>> into account in deciding whether or not backporting the name changes was
>>> a
>>> good idea.
>>>
>>> I think it's pretty clear that the code needs to be moved back into the
>>> modules with the old names for 2.6. The only question is whether or not
>>> we
>>> put any effort into making the new stdlib organisation usable in 2.x, or
>>> just rely on 2to3 to fix it (note that the "increasing the common subset"
>>> argument doesn't really apply, since you can catch the import errors in
>>> order to try both names).
>>
>> Problem with this is it makes forward-porting revisions to 3.0 a PITA.
>> By keeping the module names consistent between the versions merging a
>> revision is just a matter of ``svnmerge merge`` with the usual
>> 3.0-specific changes. Reverting the modules back to the old name will
>> make forward-porting much more difficult as I don't think svn keeps
>> rename information around (and thus map the old name to the new name
>> in terms of diffs).
>
> svnmerge is written in Python, so wouldn't it be possible to add
> support for maintaining such renaming to that tool ?
>

Don't know, possibly. But I am not about to try to figure out.

> I don't think that an administrative problem such as forward-
> porting patches to 3.x warrants breakage in the 2.x branch.
>

That's why I suggested changing pickle to deal with the rename, but
obviously I am in the minority in that idea.

> After all, the renaming was approached for Python 3.0 and not
> 2.6 *because* it introduces major breakage.
>
> AFAIR, the discussion on the stdlib-sig also didn't include the
> plan to backport such changes to 2.6. Otherwise, we would have
> hashed them out there.
>

Never came up.

>> Alexandre's idea of teaching pickle the mapping of old names to new
>> might be the best solution. We could have a flag to pickle that
>> deactivates the renaming. Otherwise we could bump the pickle version
>> number so that the new number doesn't do the mapping while the old
>> versions to the implicit module mapping.
>>
>> And as Greg and Glpyh have pointed out, this is a problem that might
>> need to be addressed in the future with some changes to our
>> serialization method (I have no clue how since I don't deal with
>> pickle very much).
>
> It is possible to make pickle aware of the module renames, but
> that doesn't solve problems with other forms of serialization
> or use of the .__module__ attribute in general.
>
> Why can't we just provide a "from __future__ import renamed_modules"
> which then provides all the new name to old name mappings in
> some form (e.g. module proxies or whatever) and leave the
> existing modules in 2.x untouched ?

I have started a discussion on the stdlib SIG on how to handle this,
so I will defer this discussion to there.

But one thing that needs to be decided is if we are ever going to
allow ourselves to rename modules without a major version bump, and if
so how to deal with this problem. I would hope we don't have to wait
another eight years before there is another chance to shift things
around if it becomes apparent that some new package should be
introduced since 2to3 gives us a very nice way to handle the
mechanical aspect of porting code.

-Brett