[issue12486] tokenize module should have a unicode API

[Matthias Bussonnier]

Matthias Bussonnier <bussonniermatthias at gmail.com> added the comment:

> Why not just bless the existing generate_tokens() function as a public API, 

Yes please, or just make the private `_tokenize` public under another name. The `tokenize.tokenize` method try to magically detect encoding which may be unnecessary.

----------
nosy: +mbussonn

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue12486>
_______________________________________