[issue3353] make built-in tokenizer available via Python C API

[Pablo Galindo Salgado]

Pablo Galindo Salgado <pablogsal at gmail.com> added the comment:

> It might also make sense to build new tokenize.py apis avoiding the `readline()` api -- I always found it painful to work with

Then we would need to maintain the old Python APIs + the new ones using the module? What you are proposing seems more than just speeding up tokenize.py re-using the existing c code

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue3353>
_______________________________________