[issue42729] tokenize, ast: No direct way to parse tokens into AST, a gap in the language processing pipiline

[Paul Sokolovsky]

Paul Sokolovsky <pfalcon at users.sourceforge.net> added the comment:

> but the thing I don't quite get is the use case.

And if that went unanswered: the usecase, how I'd formulate it, is to not expose CPython historical implementation detail of "tokenize" being disconnected from the rest of the language processing infrastructure, and make them learn tricks like needing to go back to character program form if they ever start to use "tokenize", but let it all integrate well into single processing pipeline.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42729>
_______________________________________