[issue17061] tokenize unconditionally emits NL after comment lines & blank lines

[Meador Inge]

Meador Inge added the comment:

The current behavior seems consistent with the lexical definition for
blank lines [1]:

"""
A logical line that contains only spaces, tabs, formfeeds and possibly a
comment, is ignored (i.e., no NEWLINE token is generated).
"""

NL and COMMENT are used for items that the CPython tokenizer
ignores (and are not really tokens).  Also, the test suite explicitly
tests for this case.

Perhaps the tokenize documentation should be updated
to say something like:

"""
NL tokens are generated when a logical line of code is continued over
multiple physical lines and for blank lines.
"""

[1] http://docs.python.org/3.4/reference/lexical_analysis.html#blank-lines

----------
type:  -> behavior

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17061>
_______________________________________