[New-bugs-announce] [issue44667] tokenize.py emits spurious NEWLINE if file ends on a comment without a newline

[Matthieu Dartiailh]

New submission from Matthieu Dartiailh <m.dartiailh at gmail.com>:

Using tokenize.py to tokenize the attached file yields:
0,0-0,0:            ENCODING       'utf-8'
1,0-1,2:            NAME           'if'
1,3-1,4:            NAME           'a'
1,4-1,5:            OP             ':'
1,5-1,7:            NEWLINE        '\r\n'
2,0-2,4:            INDENT         '    '
2,4-2,5:            NAME           'b'
2,6-2,7:            OP             '='
2,8-2,9:            NUMBER         '1'
2,9-2,11:           NEWLINE        '\r\n'
3,0-3,2:            NL             '\r\n'
4,0-4,6:            COMMENT        '# test'
4,6-4,6:            NL             ''
4,6-4,7:            NEWLINE        ''
5,0-5,0:            DEDENT         ''
5,0-5,0:            ENDMARKER      ''

This output is wrong in that it adds 2 newlines one as a NL which is a correct and one as a NEWLINE which is not since there is no preceding code.

If a new line is added at the end of the file, one gets:

0,0-0,0:            ENCODING       'utf-8'
1,0-1,2:            NAME           'if'
1,3-1,4:            NAME           'a'
1,4-1,5:            OP             ':'
1,5-1,7:            NEWLINE        '\r\n'
2,0-2,4:            INDENT         '    '
2,4-2,5:            NAME           'b'
2,6-2,7:            OP             '='
2,8-2,9:            NUMBER         '1'
2,9-2,11:           NEWLINE        '\r\n'
3,0-3,2:            NL             '\r\n'
4,0-4,6:            COMMENT        '# test'
4,6-4,8:            NL             '\r\n'
5,0-5,0:            DEDENT         ''
5,0-5,0:            ENDMARKER      ''

Similarly if code is added before the comment, a single NEWLINE is generated (with no text since it is fake).

The extra NEWLINE found when tokenizing the attached file can cause issue when parsing the file. It was found in https://github.com/we-like-parsers/pegen/pull/11#issuecomment-881926767 where a pure python parser based on pegen is being built. The extra NEWLINE is an issue since the grammar does not accept NEWLINE at the end of a block and cause parsing to fail using the same rules as the python grammar while the cpython parser can handle this file without any issue.

I believe this issue stems from https://github.com/python/cpython/blob/3.9/Lib/tokenize.py#L605 where the check does not account for a last line limited to comments. Adding a check to determine if the line starts with a # should be sufficient to avoid emitting the extra NEWLINE.

----------
components: Library (Lib)
files: no_newline_at_end_of_file_with_comment.py
messages: 397750
nosy: mdartiailh
priority: normal
severity: normal
status: open
title: tokenize.py emits spurious NEWLINE if file ends on a comment without a newline
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file50157/no_newline_at_end_of_file_with_comment.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue44667>
_______________________________________