[issue12486] tokenize module should have a unicode API

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

The compiler has a PyCF_SOURCE_IS_UTF8 flag: see compile() builtin. The parser has a flag to ignore the coding cookie: PyPARSE_IGNORE_COOKIE.

Patch tokenize to support Unicode is simple: use PyCF_SOURCE_IS_UTF8 and/or PyPARSE_IGNORE_COOKIE flags and encode the strings to UTF-8.

Rewrite the parser to work directly on Unicode is much more complex and I don't think that we need that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12486>
_______________________________________