[issue25311] Add f-string support to tokenize.py

[Martin Panter]

Martin Panter added the comment:

I was just about to make the same bug report :) I guess it would be fine to tokenize F-strings as the same string objects as others, it probably just needs adding an F to the right regular expression.

$ ./python -btWall -m tokenize
"string"
1,0-1,8:            STRING         '"string"'     
1,8-1,9:            NEWLINE        '\n'           
b"string"
3,0-3,9:            STRING         'b"string"'    
3,9-3,10:           NEWLINE        '\n'           
f"string"
4,0-4,1:            NAME           'f'            
4,1-4,9:            STRING         '"string"'     
4,9-4,10:           NEWLINE        '\n'

----------
keywords: +easy
nosy: +martin.panter

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25311>
_______________________________________