[Python-Dev] Pre-existing bug of the millennium (so far)

[Tim Peters]

This one came up while working on ZODB:

    weakref callback vs. gc vs. threads
    http://www.python.org/sf/1055820

Short course:  in the presence of weakrefs, cyclic gc is still hosed (it
turns out that neither threads nor weakref callbacks are necessary to get
hosed).

temp2a.py there demonstrates there's a problem, but in an unclear way
(hundreds of objects, hundreds of weakrefs and weakref callbacks (all via
WeakValueDictionary internals), 3 threads).  OTOH, there's nothing clever or
tricky about it.  Sooner or later, it just fails (an accessible instance of
a user-defined class gets its __dict__ cleared "by magic").

temp2b.py reduces it to 2 objects and 1 thread.  This is contrived, but is
deterministic.

temp2c.py boosts it to 3 objects, and is a nightmare:  it shows that the
problem can occur during a gc collection that doesn't see *any* objects
having a weakref with a callback.   There is a weakref with a callback here,
but it's attached to an object in an older generation, and collection of a
younger generation triggers that callback indirectly.  Because this is such
a nasty case (no amount of analysis of the objects in the generation being
collected can deduce that it's possible for a weakref callback to run),
there are extensive comments and an ASCII-art diagram in the file.

Even worse, temp2d.py shows we can get in trouble even if there's never a
weakref with a callback.  It's enough to have one weakref (without a
callback), and one object with a __del__ method.

Offhand, I don't have a plausible solution that will work.  The elegant
<wink> analysis in gc_weakref.txt missed what should have been obvious even
then:  cyclic trash is still potentially reachable via "global" weakrefs, so
if any Python code whatsoever can run while gc is breaking cycles (whether
via __del__ or via wr callback), global weakrefs can resurrect cyclic trash.
That suggests some Draconian approaches.

Anyone have a bright idea?  It's remarkable how long we've managed to go
without noticing that everything is disastrously broken here <0.9 wink>.