[New-bugs-announce] [issue35766] Merge typed_ast back into CPython

[Guido van Rossum]

New submission from Guido van Rossum <guido at python.org>:

(This started at https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377. It's somewhat related to https://bugs.python.org/issue33337.) 

I now have a thorough understanding of what typed_ast does, and I think it would be straightforward to port it upstream. We’d need to define two new tokens to represent `# type: ignore` and `# type: <whatever>`, and tokenizer code to recognize these. Then we need a new flag to be passed to the tokenizer (via the parser) that enables this behavior. We make a small number of changes to `Grammar` (inserting optional `TYPE_COMMENT` tokens and to `Python.asdl` (adding fields to a few node types to hold the optional type comment), and a fair number of changes to `ast.c` to extract the type comments. We have similar patches for 3.6 and 3.7, so it's a simple matter of porting those patches to 3.8.

By default, `ast.parse()` should not return type comments, since this would reject some perfectly good Python code (with sonething looking like a type comment in a place where the grammar doesn’t allow it). But passing an new flag will cause the tokenizer to process type comments and the returned tree will contain them.

I could produce a PR with this in a few days (having just gone over most of the process for porting typed_ast from 3.6 to 3.7).

There’s one more feature I’d like to lobby for – a feature_version flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4). This is used in mypy to decouple the Python version you’re running from the Python version for which you’re checking compatibility (useful when checking code that will be deployed on a system with a different Python version installed). I imagine this would be useful to other linters as well, and the implementation is mostly manipulating whether `async` and `await` are keywords. But if there’s pushback to this part I can live without it – the rest of the work is still useful.

In the next few days I will produce a PR so people can see for themselves.

In https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/17, Łukasz offered to merge my PR.

----------
components: Library (Lib)
messages: 333919
nosy: gvanrossum
priority: normal
severity: normal
stage: needs patch
status: open
title: Merge typed_ast back into CPython
type: enhancement
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35766>
_______________________________________