[Python-Dev] PEP 511 (code transformers) rejected

[Nick Coghlan]

On 2 November 2017 at 23:42, Victor Stinner <victor.stinner at gmail.com> wrote:
> (Email resent, I first sent it to Nick privately by mistake.)

Oops, I didn't even notice that. Reposting my reply below.

> 2017-11-02 2:53 GMT+01:00 Nick Coghlan <ncoghlan at gmail.com>:
>> The piece that we're currently missing to make such workflows easier to
>> manage is an equivalent of JavaScript's source maps (...)
>
> Code objects already have a instruction pointer => line number mapping
> table: the code.co_lnotab field. It's documented at:
> https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt
>
> This table is built from the line number information of the AST tree.
>
> The mapping table is optimized to be small. Before Python 3.5, line
> number had to be monotonic. Since Python 3.6, you "move" instructions
> at the AST level, and so have "non-monotonic" line numbers (ex: line
> 1, line 3, line 2, line 4).

Right, and linecache knows how to read that, However, it can only do
so if the source files are on the running system with the bytecode
files, *and* the source code we're interested in is the source code
that was actually compiled by the interpreter.

Source code transformers fail that second precondition (since the
interpreter only sees the post-transformation code), and this is one
of the big reasons folks ended up writing actual single source 2/3
compatible code bases rather than running 2to3 as a source code
transformer when building packages: with transformed source,
conventional tracebacks quote the line from the transformed source
code, *not* the line in the original pre-transformation source code.

However, if the code transformer were to emit a JavaScript style
source map in addition to emitting the transformed code, then
automated tooling could take a traceback that referenced lines in the
transformed code, and work out the equivalent traceback for the
pre-transformation code. (I believe Cython has something like that in
order to provide it's HTML annotation mode, and PyPy's JIT can trace
from machine code back to the related Python source lines, but we
don't have anything that's independent of a particular toolchain the
way source maps are)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia