[issue3353] make built-in tokenizer available via Python C API

[Fredrik Lundh]

Fredrik Lundh <effbot at users.sourceforge.net> added the comment:

There are a few things in the struct that needs to be public, but that's
nothing that cannot be handled by documentation.  No need to complicate
the API just in case.

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3353>
_______________________________________