[Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523

[Terry Reedy]

On 3/25/2017 8:56 AM, Serhiy Storchaka wrote:
> On 25.03.17 12:04, Victor Stinner wrote:
>> https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/
>>
>>
>> "What changed in Python 3.6 to allow this?
>>
>> The new frame evaluation API was introduced to CPython in PEP 523 and it
>> allows to specify a per-interpreter function pointer to handle the
>> evaluation of frames."
>>
>> Nice!
>
> Awesome! Any chance that pdb can utilize similar technique? Or this
> doesn't make sense for pdb?

According to the bdb.Bdb docstring, pdb implements a command-line user 
interface on top of bdb, while  bdb.Bdb "takes care of the details of 
the trace facility".  idlelib.debugger similarly implements a GUI user 
interface on top of bdb.  I am sure that there are other debuggers that 
build directly or indirectly (via pdb) on bdb.  So the question is 
whether bdb can be enhanced or augmented with a C-coded _bdb or other 
new module.

As I understand it, sys.settrace results in an execution break and 
function call at each point in the bytecode corresponding to the 
beginning of a (logical?) line.  This add much overhead.  In return, a 
trace-based debugger allows one to flexibly control stop and go 
execution either with preset breakpoints* or with interactive commands: 
step (one line), step into (a function frame), step over (a function 
frame), or go to next breakpoint.  The last is implemented by the 
debugger automatically stepping at each break call unless the line is in 
the existing breakpoint list.

* Breakpoints can be defined either in an associated editor or with 
breakpoint commands in the debugger when execution is stopped.

PEP 523 envisioned an alternate non-trace implementation of 'go to next 
breakpoint' by a debugger going "as far as to dynamically rewrite 
bytecode prior to execution to inject e.g. breakpoints in the bytecode."
https://www.python.org/dev/peps/pep-0523/#debugging

A debugger doing this could either eliminate the other 'go' commands 
(easiest) or implement them by either setting temporary breakpoints or 
temporarily turning tracing on.

I presume it should be possible to make bdb.Bdb use bytecode breakpoints 
or add a new class with a similar API.  Then any bdb-based debugger to 
be modified to make the speedup available.

-- 
Terry Jan Reedy