[issue9969] tokenize: add support for tokenizing 'str' objects

[Martin Panter]

Martin Panter added the comment:

I left some comments. Also, it would be nice to use the new function in the documentation example, which currently suggests tunnelling through UTF-8 but not adding an encoding comment. And see the patch for Issue 12486, which highlights a couple of other places that would benefit from this function.

----------
nosy: +martin.panter

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9969>
_______________________________________