[Python-Dev] Combining the best of PEP 288 and PEP 325: generator exceptions and cleanup

[Tim Peters]

[Phillip J. Eby]
> ...
> However, Tim's new post brings up a different issue: if the collector can't
> tell the difference between a cycle participant and an object that's only
> reachable from a cycle, then the mere existence of a generator __del__ will
> prevent the cycle collection of the entire traceback/frame system that
> includes a generator-iterator reference anywhere!  And that's a pretty
> serious problem.

It's not that simple <wink>.  If an object with a __del__ is not part
of a cycle, but is reachable only from trash cycles, that __del__ does
not inhibit garbage collection.  Like:

    A<->B -> C -> D -> E

where C, D and E have __del__, but A and B don't, and all are trash.

Relatively early on, gc "moves" C, D, and E into a special
"finalizers" list, and doesn't look at this list again until near the
end.  Then A.tp_clear() and B.tp_clear() are called in some order.  As
a *side effect* of calling B.tp_clear(), C's refcount falls to 0, and
Python's normal refcount-based reclamation (probably) recovers all of
C, D and E, and runs their __del__ methods.  Note that refcount-based
reclamation necessarily follows a DAG order:  E is still intact when
D.__del__ is called, and likewise D is still intact when C.__del__ is
called.  It's possible that C.__del__ will resurrect D and/or E, and
that D.__del__ will resurrect E.  In such cases, D and/or E's
refcounts don't fall to 0, and their __del__ methods won't be called
then.

Cyclic gc doesn't force any of that, though -- it's all a side effect
of the clear() in gcmodule.c's:

		if ((clear = op->ob_type->tp_clear) != NULL) {
			Py_INCREF(op);
			clear(op);
			Py_DECREF(op);

In turn, one of A and B get reclaimed as a side effect of the
Py_DECREF there -- it's one of the delights of gcmodule.c that if you
don't know the trick, you can stare at it for hours and never discover
where exactly it is anything gets released <0.9 wink>.  In fact, it
doesn't release anything directly -- "all it really does" now is break
reference cycles, so that Py_DECREF can do its end-of-life thing.