[issue35107] untokenize() fails on tokenize output when a newline is missing

[Ammar Askar]

Ammar Askar <ammar at ammaraskar.com> added the comment:

fwiw I think there's more at play here than the newline change. This is the behavior I get on 3.6.5 (before the newline change is applied). # works as expected but check out this input:

>>> t.untokenize(tokenize.generate_tokens(io.StringIO('#').readline))
'#'
>>> t.untokenize(tokenize.generate_tokens(io.StringIO('x=1').readline))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Python365\lib\tokenize.py", line 272, in untokenize
    self.add_whitespace(start)
  File "D:\Python365\lib\tokenize.py", line 234, in add_whitespace
    .format(row, col, self.prev_row, self.prev_col))
ValueError: start (1,0) precedes previous end (2,0)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35107>
_______________________________________