[Python-Dev] pickle and copy discrepancy

[Serhiy Storchaka]

The pickle and the copy modules use the same protocol. The reconstruct 
the object by data returned by the __reduce_ex__/__reduce__ method, but 
does it in different and incompatible way.

In general case the result of __reduce__ includes:

1) The class of the object and arguments to __new__().
2) The state passed to __setstate__() (or a dict of attributes and 
possible a tuple of __slots__ values).
3) An iterator of list items that should be appended to the object by 
calling extend() or append().
4) An iterator of key-value value pairs that should be set in the object 
by calling update() or __setitem__().

The difference is that the copy module sets object's state before adding 
items and key-value pairs, but the pickle module sets object's state 
after adding items and key-value pairs. If append() or __setitem__() 
depend on the state of the object, the pickling is incompatible with the 
copying.

The behaviour of copy was changed in issue1100562 [1] (see also 
issue1099746 [2]). But this caused a problem with other classes (see 
issue10131 [3]). Changing either pickle or copy for sure will break 
existing code. But keeping current discrepancy makes existing code not 
correct and makes too hard to write correct code that works with both 
pickle and copy. For sure most existing code for which it is matter is 
not correct.

The behaviour of default reducing method for dicts/lists subclasses is 
not documented [4].

We should choose in what direction we have to break backward 
compatibility. The behaviour of the copy module looks more natural. It 
allows to work correctly most naive implementations (as in [2]). The 
pickle module is more used and breaking it can cause more harm. But the 
order of object reconstruction is determined at pickling time, thus 
already pickled data will be reconstructed with old order. The change 
will only affect new pickles.

[1] http://bugs.python.org/issue1100562
[2] http://bugs.python.org/issue1099746
[3] http://bugs.python.org/issue10131
[4] http://bugs.python.org/issue4712