[New-bugs-announce] [issue25079] Tokenize generates NL instead of NEWLINE for comments

[Optimal BPM]

New submission from Optimal BPM:

Quoting the documentation:

tokenize.NL
Token value used to indicate a non-terminating newline. The NEWLINE token indicates the end of a logical line of Python code; NL tokens are generated when a logical line of code is continued over multiple physical lines.

This doesn't seem to be entirely true.
At the end of comments, an .NL, not .NEWLINE is generated:
TokenInfo(type=55 (NL), string='\n', start=(5, 22), end=(5, 23), line='# Some docs for the IF\n')

As a comment cannot extend over several lines, if would appear that a NEWLINE would be appropriate?

----------
components: Interpreter Core
messages: 250516
nosy: Optimal BPM
priority: normal
severity: normal
status: open
title: Tokenize generates NL instead of NEWLINE for comments
type: behavior
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25079>
_______________________________________