[New-bugs-announce] [issue25555] Fix parser and AST: fill lineno and col_offset when compiling AST from Python objects

[STINNER Victor]

New submission from STINNER Victor:

I wrote an AST optimizer which build AST from source code using ast.parse(), modify AST in place, and then compile the AST to bytecode.

The problem is that the lineno and col_offset attributes of the "arg" AST node are uninitialized when AST is rebuild from Python objects: compile(ast_tree, ...). As a consequence, the compilation may fail because lineno or col_offset values are invalid.

Attached patch enhances Parser/asdl_c.py to take "arg" attributes in account. It also updates the generated files Include/Python-ast.h and Python/Python-ast.c, and fix usage of the arg() function in Python/ast.c.

It looks like only the "arg" AST node had attributes which were initialized (other AST nodes are handled with a different code path which already filled attributes).

See Parser/Python.asdl for the definition of the "arg" AST node.

Note: Python 2.7 is not affected. In Python 2, function arguments are simple expressions, they don't have a dedicated type with lineno and col_offset attributes. "arg" was introduced in Python 3 with the PEP 3107 (function annotations).

----------
files: ast.patch
keywords: patch
messages: 254093
nosy: benjamin.peterson, haypo, matrixise, yselivanov
priority: normal
severity: normal
status: open
title: Fix parser and AST: fill lineno and col_offset when compiling AST from Python objects
versions: Python 3.5, Python 3.6
Added file: http://bugs.python.org/file40949/ast.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25555>
_______________________________________