[issue35975] Put back the ability to parse files where async/await aren't keywords

[Guido van Rossum]

New submission from Guido van Rossum <guido at python.org>:

Now that the ast module can be used to parse type comments, there’s one more feature I’d like to lobby for – a flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4).

This is used in mypy to decouple the Python version you’re running from the Python version for which you’re checking compatibility (useful when checking code that will be deployed on a system with a different Python version installed). I imagine this would be useful to other linters as well, and the implementation is mostly manipulating whether `async` and `await` are keywords. But if there’s pushback to this part I can live without it – the rest of the work is still useful.

The implementation in typed_ast takes the form of a feature_version flag which is set to the minor Python version to be parsed. Setting it to 4 or lower would make async/await never keywords, setting it to 5 or 6 would make them conditional per PEP 492, and setting it to 7 or higher would make these unconditional keywords. A few minor uses of the same flag also reject f-strings, annotated assignments (PEP 526), and matrix multiply for versions where those don't exist.

But I don't need all of those -- I really just need to be able to select the 3.5/3.6 rules for conditional async/await keywords, since all the other features are purely backwards compatible, whereas with async/await there is legal (and plentiful!) code that uses these as variable or function names that should be supported in 3.5/3.6 mode.

Of course having it be a (minor) version number would still be more future-proof -- if say in 3.10 we add a match expression using some kind of conditional keyword hack, we might have to dust it off.

Note that much of what I'm asking for would effectively roll back https://bugs.python.org/issue30406 -- sorry Jelle! (Though there's more to it -- Serhiy's introduction of Grammar/Token followed, and I would still need to thread some kind of flag all the way from ast.parse() to tokenizer.c.

----------
messages: 335277
nosy: gvanrossum
priority: normal
severity: normal
status: open
title: Put back the ability to parse files where async/await aren't keywords

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35975>
_______________________________________