[New-bugs-announce] [issue21385] Compiling modified AST crashes on debug build unless linenumbering discarded

[Antti Haapala]

New submission from Antti Haapala:

We had had problems with our web service occasionally hanging and performing poorly, and as we didn't have much clue about the cause of these, we decided to continuously run our staging build under debug enabled python 3.4, and then attaching gdb as needed. To much dismay we found out that our code generating code that builds AST trees and then compiles them to modules is dumping cores on the debug version. 

The assertion is the much discussed "linenumbers must grow monotonically" at http://hg.python.org/cpython/file/04f714765c13/Python/compile.c#l3969

In our case, the AST is generated from a HTML template with embedded python parts; as we could approximately point out much of the corresponding code in the source template, we decided to reuse the linenumbers in AST, and things seemed to work quite nicely and usually we could get usable tracebacks too.

Under debug build, however, as the ordering of some constructs in the source language are different from python, we need to discard *all* linenumbers and only after then use fix_missing_locations, and thus get completely unusable traces from these parts of code, all happening on line 1. Just using fix_missing_locations does not work. Likewise the rules for which parts of the tree should come in which order in the lnotab is quite hard to deduce.

It seems to me that when the lnotab was created, no one even had in mind that there would be an actually useful AST module that would be used for code generation. Considering that there have been other calls for breaking the correspondence of bytecode addresses to monotonically growing linenumbers, I want to reopen the discussion about changing the lnotab structures now to allow arbitrary mapping of source code locations to bytecode, and especially about the need for this assertion in the debug builds at all.

Attached is an example of code that appends a function to an existing module syntax tree, run under python*-dbg it dumps the core with "Python/compile.c:nnnn: assemble_lnotab: Assertion `d_lineno >= 0' failed." Ofc in this simple case it is easy to just modify the linenumbers so that function "bar" would come after "foo", however in some cases it is hard to know the actual rules; fix_missing_locations does not do this right at all.

I am also pretty sure most of the existing code that combine parsed and generated ASTs and then compile the resulting trees also would fail that assert, but no one is ever running their code under debug builds.

----------
components: Interpreter Core
files: astlinenotest.py
messages: 217502
nosy: Ztane
priority: normal
severity: normal
status: open
title: Compiling modified AST crashes on debug build unless linenumbering discarded
type: crash
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file35090/astlinenotest.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21385>
_______________________________________