[issue12486] tokenize module should have a unicode API

[Martin Panter]

Martin Panter added the comment:

I didn’t notice that this dual untokenize() behaviour already existed. Taking that into account weakens my argument for having separate text and bytes tokenize() functions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12486>
_______________________________________