[Python-ideas] Compressing excepthook output

[אלעזר]

I suggest adding an excepthook that prints out a compressed version of the
stack trace. The new excepthook should be the default at least for
interactive mode.

The use case is this: you are using an interactive interpreter, or perhaps
in eclipse's PyDev, experimenting with some code. The code happen to has an
infinite recursion - maybe an erroneous boundary condition, maybe the
recursion itself was an accident. You didn't catch the RuntimeError so you
get a print of the traceback. This is by default a 2000 lines of highly
repetitive call chain. Most likely, a single cycle repeating some 300 times.

The main problem is that in most environments, by default, you have only a
limited amount of lines kept in the window. So you can't just scroll up and
see what was the error in the first place - where is the entry point into
the cycle. You have to reproduce it, and catch RuntimeError. You can't just
use printing for debugging, either, because you won't see them. And even if
you can see it, you had lost much of your "history" for nothing.

I have tried to implement an alternative for sys.excepthook (see below),
which compresses the last simple cycle in the call graph. Turns out it's
not trivial, since the traceback object is not well documented (and maybe
it shouldn't be, as it is an implementation detail) so it's non trivial (if
at all possible) to change the trace list in an existing traceback. I don't
think it is reasonable to just send anyone interested in such a feature to
implement it themselves - especially given that newcomers ate its main
target - and even if we do, there is no simple way to make it a default.

Such a compression will not always help, since the call graph may be
arbitrarily complex, so there has to be some threshold below which there
won't be any compression. this threshold should be chosen after considering
the number of lines accessible by default in common environments
(Linux/Windows terminals, eclipse's console, etc.).
Needless to say, the output should be correct in all cases. I am not sure
that my example implementation is.

Another suggestion, related but distinct: `class
RecursionLimitError(RuntimeError)` should be raised instead of a plain
RuntimeError. One should be able to except this specific case,
and "Exception messages are not part of the Python API".

---

Example for the desired result (non interactive):

Traceback (most recent call last):
  File "/workspace/compress.py", line 48, in <module>
    bar()
  File "/workspace/compress.py", line 46, in bar
    p()
  File "/workspace/compress.py", line 43, in p
    def p(): p0()
  File "/workspace/compress.py", line 41, in p0
    def p0(): p2()
  File "/workspace/compress.py", line 39, in p2
    def p2(): p()
RuntimeError: maximum recursion depth exceeded
332.67 occurrences of cycle of size 3 detected

Code:

import traceback
import sys

def print_exception(name, value, count, size, newtrace):
    # this is ugly and fragile
    sys.stderr.write('Traceback (most recent call last):\n')
    sys.stderr.writelines(traceback.format_list(newtrace))
    sys.stderr.write('{}: {}\n'.format(name ,value))
    sys.stderr.write('{} occurrences of cycle of size {}
detected\n'.format(count, size))

def analyze_cycles(tb):
    calls = set()
    size = 0
    for i, call in enumerate(reversed(tb)):
        if size == 0:
            calls.add(call)
            if call == tb[-1]:
                size = i
        elif call not in calls:
            length = i
            break
    return size, length

def cycle_detect_excepthook(exctype, value, trace):
    if exctype is RuntimeError:
        tb = traceback.extract_tb(trace)
        # Feels like a hack here
        if len(tb) >= sys.getrecursionlimit()-1:
            size, length = analyze_cycles(tb)
            count = round(length/size, 2)
            if count >= 2:
                print_exception(exctype.__name__, value, count, size,
tb[:-length+size])
                return
    sys.__excepthook__(exctype, value, tb)

sys.excepthook = cycle_detect_excepthook

if __name__ == '__main__':
    def p2(): p()

    def p0(): p2()

    def p(): p()

    def bar():
        p()

    bar()

###

Elazar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130916/24d5e094/attachment-0001.html>