[Python-Dev] __traceback__ and reference cycles

[Tim Peters]

[Armin Rigo]
> There are various proposals to add an attribute on exception instances
> to store the traceback (see PEP 344).  A detail not discussed, which I
> thought of historical interest only, is that today's exceptions try very
> hard to avoid reference cycles, in particular the cycle
>
>   'frame -> local variable -> traceback object -> frame'
>
> which was important for pre-GC versions of Python.  A clause 'except
> Exception, e' would not create a local reference to the traceback, only
> to the exception instance.  If the latter grows a __traceback__
> attribute, it is no longer true, and every such except clause typically
> creates a cycle.
>
> Of course, we don't care, we have a GC -- do we?  Well, there are cases
> where we do: see the attached program...  In my opinion it should be
> considered a bug of today's Python that this program leaks memory very
> fast and takes longer and longer to run each loop (each loop takes half
> a second longer than the previous one!).  (I don't know how this bug
> could be fixed, though.)
>
> Spoiling the fun of figuring out what is going on, the reason is that
> 'e_tb' creates a reference cycle involving the frame of __del__, which
> keeps a reference to 'self' alive.  Python thinks 'self' was
> resurrected.  The next time the GC runs, the cycle disappears, and the
> refcount of 'self' drops to zero again, calling __del__ again -- which
> gets resurrected again by a new cycle.  Etc...  Note that no cycle
> actually contains 'self'; they just point to 'self'.  In summary, no X
> instance gets ever freed, but they all have their destructors called
> over and over again.
>
> Attaching a __traceback__ will only make this "bug" show up more often,
> as the 'except Exception, e' line in a __del__() method would be enough
> to trigger it.
>
> Not sure what to do about it.  I just thought I should share these
> thoughts (I stumbled over almost this problem in PyPy).

I can't think of a Python feature with a higher aggregate
braincell_burned / benefit ratio than __del__ methods.  If P3K retains
them-- or maybe even before --we should consider taking "the Java
dodge" on this one.  That is, decree that henceforth a __del__ method
will get invoked by magic at most once on any given object O, no
matter how often O is resurrected.

It's been mentioned before, but it's at least theoretically
backward-incompatible, so "it's scary".  I can guarantee I don't have
any code that would care, including all the ZODB code I watch over
these days.  For ZODB it's especially easy to be sure of this:  the
only __del__ method in the whole thing appears in the test suite,
verifying that ZODB's object cache no longer gets into an infinite
loop when a user-defined persistent object has a brain-dead __del__
method that reloads self from the database.  (Interestingly enough, if
Python guaranteed to call __del__ at most once, the infinite loop in
ZODB's object cache never would have appeared in this case.)