[issue3353] make built-in tokenizer available via Python C API

[Dustin J. Mitchell]

Dustin J. Mitchell added the comment:

>From my read of this bug, there are two distinct tasks mentioned:

 1. make PyTokenizer_* part of the Python-level API
 2. re-implement 'tokenize' in terms of that Python-level API

#1 is largely complete in Andrew's latest patch, but that will likely need:
 * rebasing
 * hiding struct fields
 * documentation

#2 is, I think, a separate project.  There may be good reasons *not* to do this which I'm not aware of, and barring such reasons the rewrite will be difficult and could potentially change behavior like issue2180.  So I would suggest filing a new issue for #2 when #1 is complete.  And I'll work on #1.

----------
nosy: +djmitche

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3353>
_______________________________________