[issue33337] Provide a supported Concrete Syntax Tree implementation in the standard library

[Łukasz Langa]

Łukasz Langa <lukasz at langa.pl> added the comment:

> But lib2to3 is proof that the stdlib is just as much subject to stalling.

The issue here is internal visibility. "lib2to3" is a library that supports "2to3" which is rather neglected internally since we started promoting `six` as a better migration strategy to Python 3.

Most core devs don't even *know* new syntax is supposed to be added to lib2to3.  Case in point: somehow Lib/tokenize.py was updated just in time for f-strings to be released but not Lib/lib2to3/pgen2/tokenize.py.

By unifying the tokenizers and moving the CST out of lib2to3's guts (and documenting it as a supported feature!), I'm pretty sure we can eliminate the danger of forgetting to update it in the future.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33337>
_______________________________________