[issue40312] Weakref callbacks running before finalizers in GC collection

[Tim Peters]

Tim Peters <tim at python.org> added the comment:

Things get complicated here because in older versions of Python an instance of ForeverObject(True) could "leak" forever:  if an object in a trash cycle had a __del__ method, that method would never be called, and the object would never be collected.

Starting in Python 3.4, that changed:  __del__ no longer inhibits collection of objects in cyclic trash.  However, __del__ is called no more than once starting in 3.4.  If an object is resurrected by __del__, it's marked with a "__del__ was already called" bit, and __del__ is never called again by magic if/when the object becomes trash again.

I don't think the weakref docs were changed, because nobody cares ;-)  

What it _intended_ to mean by "about to be finalized" is clear as mud.  What it actually means is akin to "about to have its memory destroyed and recycled".

In current CPython, for your ForeverObject(False), `del o` does not make the object trash "for real".  __del__ runs immediately (due to deterministic, synchronous reference counting) and resurrects it.  That cuts off the "about to have its memory destroyed and recycled" part, so the callback doesn't run.

But if you do

    del o

again, _then_ the callback runs.  __del__ isn't run again, so the object isn't resurrected again, so the "about to have its memory destroyed and recycled" part applies.

In cyclic gc, there is no deterministic order in which end-of-life actions occur.  There may well be thousands of objects in cyclic trash, or reachable only from cyclic trash.  The order picked is more-or-less arbitrary, just trying like hell to ensure that no end-of-life action ever "sees" an object whose memory has already been destroyed and recycled.

To make progress at all, it _assumes_ all the cyclic trash really will be reclaimed (memory destroyed and recycled).  That's why it runs all weakref callbacks to trash objects (provided the weakref isn't also trash).  It also runs all finalizers (except on objects with a __del__ that has already been called).  Only after _all_ that is done does it even start to destroy and recycle memory.

Although, along the way, memory _may_ be destroyed and recycled as a result of refcounts falling to 0 as end-of-life actions (callbacks and finalizers) are invoked.

And, yup, it's as delicate as it sounds ;-)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40312>
_______________________________________