[issue25079] Tokenize generates NL instead of NEWLINE for comments
Optimal BPM
report at bugs.python.org
Sat Sep 12 11:49:19 CEST 2015
New submission from Optimal BPM:
Quoting the documentation:
tokenize.NL
Token value used to indicate a non-terminating newline. The NEWLINE token indicates the end of a logical line of Python code; NL tokens are generated when a logical line of code is continued over multiple physical lines.
This doesn't seem to be entirely true.
At the end of comments, an .NL, not .NEWLINE is generated:
TokenInfo(type=55 (NL), string='\n', start=(5, 22), end=(5, 23), line='# Some docs for the IF\n')
As a comment cannot extend over several lines, if would appear that a NEWLINE would be appropriate?
----------
components: Interpreter Core
messages: 250516
nosy: Optimal BPM
priority: normal
severity: normal
status: open
title: Tokenize generates NL instead of NEWLINE for comments
type: behavior
versions: Python 3.4
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25079>
_______________________________________
More information about the Python-bugs-list
mailing list