[Python-Dev] Evil reference cycles caused Exception.__traceback__

[Nick Coghlan]

On 18 September 2017 at 20:52, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 18 Sep 2017 20:35:02 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Rather than being thread local or context local state, whether or not
>> to keep the full frames in the traceback could be a yet another
>> setting on the *exception* object, whereby we tweaked the logic that
>> drops the reference at the end of an except clause as follows:
>
> That doesn't solve the problem, since the issue is that exceptions can
> be raised (and then silenced) in many places, and you don't want such
> exception-raising code (such as socket.create_connection) to start
> having to set an option on the exceptions it raises.

Bleh, in trying to explain why my proposal would be sufficient to
break the problematic cycles, I realised I was wrong: if we restrict
the frame clearing to already terminated frames (as would be necessary
to avoid breaking any still executing functions), then that means we
won't clear the frame running the exception handler, and that's the
frame that creates the problematic cyclic reference.

However, I still think it makes more sense to focus on the semantics
of preserving an exception beyond the life of the stack being unwound,
rather than on the semantics of raising the exception in the first
place.

In the usual case, the traceback does keep the whole stack alive while
the stack is being unwound, but then the exception gets thrown away at
the end when sys.exc_info() gets reset back to (None, None, None), and
then all the frames still get cleaned up fairly promptly (this is also
the case in Python 2).

We only get problems when one of the exception handlers in the stack
grabs an additional reference to either the traceback or the exception
and hence creates a persistent cyclic reference from one of the frames
back to itself. The difference in Python 3 is that saving the
exception is reasonably common, while explicitly saving the traceback
is relatively rare, so the "exc.__traceback__" is keeping tracebacks
alive that would otherwise have been cleaned up more
deterministically.

Putting the problem that way gave me an idea, though: what if, when
the interpreter was setting "sys.exc_info()" back to (None, None,
None) (or otherwise dropping an exception instance from being the
"currently active exception") it automatically set exc.__traceback__
to None?

That way, if you wanted the *traceback* (rather than just the
exception) to live beyond the stack being unwound, you'd have to
preserve the entire sys.exc_info() triple (or at least save
"exc.__traceback__" separately from "exc"). By doing that, we'd have
the opportunity to encourage folks that are considering preserving the
entire traceback to extract a TracebackException instead and some
themselves from some potentially nasty reference management issues:
https://docs.python.org/3/library/traceback.html#tracebackexception-objects

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia