[issue40246] Different error messages for same error - invalid string prefixes

[Lysandros Nikolaou]

Lysandros Nikolaou <lisandrosnik at gmail.com> added the comment:

At the moment, if the prefix is invalid, it gets tokenized as a NAME node and the string following is tokenized as a STRING node with no prefix.

╰─ echo "ur''" | ./python -m tokenize
1,0-1,2:            NAME           'ur'           
1,2-1,4:            STRING         "''"           
1,4-1,5:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''

╰─ echo "f''" | ./python -m tokenize  
1,0-1,3:            STRING         "f''"          
1,3-1,4:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue40246>
_______________________________________