[issue30377] Unnecessary complexity in tokenize.py around handling of comments and newlines

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The code is correct, it just can be made cleaner. No need to backport the change to other versions.

The line "if line[pos] in '#\r\n':" looks a kind of optimization. In common case (not a newline and not a comment) there is only one check. The expression "(NL, COMMENT)[line[pos] == '#']" is redundant of course, it can be replaced by just "NL". Note that now the line that yields yield TokenInfo(NL, ...) is almost the same for comments and newlines. If rename nl_pos to pos the same line can be used in both cases.

----------
nosy: +serhiy.storchaka
versions:  -Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue30377>
_______________________________________