[issue3353] make built-in tokenizer available via Python C API

[Pablo Galindo Salgado]

Pablo Galindo Salgado <pablogsal at gmail.com> added the comment:

> Either works for me, would you be able to point me to the starting bits as to how `_ast` becomes builtin?


https://github.com/python/cpython/blob/master/Python/Python-ast.c#L10075-L10079 

and 

https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/PC/config.c#L84

But before that I have some questions. For example: How do you plan to implement the readline() interface that tokenize.py uses in the c-module without modifying tokenize.c?

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue3353>
_______________________________________