[issue40334] PEP 617: new PEG-based parser

[Serhiy Storchaka]

Serhiy Storchaka <storchaka+cpython at gmail.com> added the comment:

The parser generator imports modules token and tokenize. It is not correct, because they are relevant to the Python version used to run the parser generator, and not to the Python version for which the parser is generated. It works currently only because there is no differences between 3.8 and 3.9, but it will fail when you add a new token or change/remove an old one.

It should either parse the correct Grammar/Tokens file, or read the content of corresponding files Lib/token.py and Lib/tokenize.py and evaluate them with eval(). See for example Tools/scripts/generate_token.py.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40334>
_______________________________________