[issue42974] tokenize reports incorrect end col offset and line string when input ends without explicit newline

[Brian Romanowski]

Brian Romanowski <romanows at gmail.com> added the comment:

Shoot, just realized that consistency isn't the important thing here, the most important thing is that the tokenizer module exactly matches the output of the Python tokenizer.  It's possible that my changes violate that constraint, I'll take a look at that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42974>
_______________________________________