[New-bugs-announce] [issue35212] Expressions with format specifiers in f-strings give wrong code position in AST

[Arminius]

New submission from Arminius <numirias at gmail.com>:

ast.parse() will give a wrong code position for an expression inside an f-string when the expression is using a format specifier.

Compare the trees of f'{a}' and f'{a:b}' :

>>> ast.parse("f'{a}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=3, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=None,
                    ),
                ],
            ),
        ),
    ],
)

>>> ast.parse("f'{a:b}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(                        col_offset=0,
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=1, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=JoinedStr(
                            lineno=1,
                            col_offset=0,
                            values=[Str(lineno=1, col_offset=0, s='b')],
                        ),
                    ),
                ],
            ),
        ),
    ],
)

In both examples, the name "a" is at the same position in the source code, however the parsed ASTs yield two different offsets,

Name(lineno=1, col_offset=3, id='a', ctx=Load())

and

Name(lineno=1, col_offset=1, id='a', ctx=Load())

.

Expected behavior: col_offset=3 for name "a" in both f-strings.

(In my specific use case, this breaks a semantic highlighting plugin for vim.)

----------
components: Interpreter Core
messages: 329672
nosy: arminius
priority: normal
severity: normal
status: open
title: Expressions with format specifiers in f-strings give wrong code position in AST
type: behavior
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35212>
_______________________________________