[issue33766] Grammar Incongruence

[Serhiy Storchaka]

Serhiy Storchaka <storchaka+cpython at gmail.com> added the comment:

NEWLINE is not a newline. It is the NEWLINE token. And it is generated at the end of file.

$ echo 'print("a");print("b")' | ./python -m tokenize
1,0-1,5:            NAME           'print'        
1,5-1,6:            OP             '('            
1,6-1,9:            STRING         '"a"'          
1,9-1,10:           OP             ')'            
1,10-1,11:          OP             ';'            
1,11-1,16:          NAME           'print'        
1,16-1,17:          OP             '('            
1,17-1,20:          STRING         '"b"'          
1,20-1,21:          OP             ')'            
1,21-1,22:          NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''

----------
nosy: +serhiy.storchaka

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33766>
_______________________________________