[issue9969] tokenize: add support for tokenizing 'str' objects

[Thomas Kluyver]

Thomas Kluyver <thomas at kluyver.me.uk> added the comment:

I've opened a PR for issue #12486, which would make the existing but undocumented 'generate_tokens' function public:

https://github.com/python/cpython/pull/6957

I agree that it would be good to design a nicer API for this, but the perfect shouldn't be the enemy of the good.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue9969>
_______________________________________