[issue25311] Add f-string support to tokenize.py

[Eric V. Smith]

Eric V. Smith added the comment:

I've fixed this particular problem, but the tokenize module definitely has some other issues. It recompiles regexes very often when it doesn't need to, it treats single- and triple-quoted strings differently (leading to some code bloat), etc. I may open another issue to address some of these problems.

And I'll be adding more tests. tokenize is still woefully under-tested.

----------
resolution:  -> fixed

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25311>
_______________________________________