[issue25311] Add f-string support to tokenize.py

[Eric V. Smith]

Eric V. Smith added the comment:

My first attempt. Many more tests are needed.

I'm going to need to spend some time trying to figure out how parts of tokenize.py actually works. I'm not sure, for example, that endpats is initialized correctly. There definitely aren't enough tests, since if I comment out parts of endpats the tests still pass.

----------
Added file: http://bugs.python.org/file40730/issue25311.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25311>
_______________________________________