[issue33899] Tokenize module does not mirror "end-of-input" is newline behavior

[Tal Einat]

Tal Einat <taleinat at gmail.com> added the comment:

This was backported since it was considered a bug, but you are right that it broke backwards compatibility, and perhaps shouldn't have been backported.

Still, with 3.6.6 and 3.7.1 now released, that ship has sailed.

We could perhaps revert this on the 2.7 branch, but I feel that reverting this change only on 2.7 would just cause even more confusion.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33899>
_______________________________________