[Patches] [ python-Patches-1062277 ] Pickle breakage with reduction of recursive structures

[SourceForge.net]

Patches item #1062277, was opened at 2004-11-08 08:06
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1062277&group_id=5470

Category: Core (C code)
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Dima Dorfman (ddorfman)
Assigned to: Nobody/Anonymous (nobody)
Summary: Pickle breakage with reduction of recursive structures

Initial Comment:
Fix problems related to reduce cycles during pickling. A
"reduce cycle" is what happens when a __reduce__
implementation returns an args that cycles back through the
object it tried to reduce. This can't work because the
unpickler has to call the constructor with those args, but
it doesn't yet have that object to be able to place inside
the args. There are two problems related to this:

  1. The standard reduce implementation for proto < 2 in
     copy_reg (_reduce_ex) doesn't correctly deal with
     recursive structures. reduce_2 in typeobject.c does the
     right thing by using listitems and dictitems. Fix
     _reduce_ex by making it do the same thing. This is okay
     for proto < 2 because those arguments are a pickler-
     side feature. Tested in test_stdreducecycle.

  2. Our pickle implementations don't check for reduce
     cycles. This is somewhat cosmetic except that stack
     overflow protection is imperfect (for cPickle), causing
     crashes, and some kinds of cycles trigger asserts (in
     pickle). Fixed in pickle and cPickle by introducing a
     reducing_now set; on entering save_reduce, the object
     id being saved must not be in that set or we've been
     called recursively while saving the callable or
     arguments. Tested in test_reduce_cycle.

This shouldn't change any semantics. That reduce shouldn't
introduce cycles through args isn't documented, but it can't
work any other way.

Possible improvement: If we want to support reducing of real
immutable containers we might have to relax the reduction
cycle test to give the cycle a chance of resolving itself
normally (by constructing a partial object to pass to the
constructor and filling it in later). I'm not sure if this
trouble is worth it just to avoid writing a
frozenset.__setstate__.

See also: http://mail.python.
org/pipermail/python-dev/2004-October/049714.html

It would be very good if someone familiar with pickle
reviewed this; I am still not very confident that I
completely understand all the issues.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1062277&group_id=5470