tokenize.untokenize adding line continuation characters

[Peter Otten]

Rotwang wrote:

> Here's something odd I've found with the tokenize module: tokenizing 'if
> x:\n    y' and then untokenizing the result adds '\\\n' to the end.
> Attempting to tokenize the result again fails because of the backslash
> continuation with nothing other than a newline after it. On the other
> hand, if the original string ends with a newline then it works fine. Can
> anyone explain why this happens?

> I'm using Python 3.4.3 on Windows 8. Copypasted from iPython:

This looks like a bug...

$ python3.4 -c 'import tokenize as t, io; 
print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n  y").readline)))'
b'if x:\n  y\\\n'

...that is fixed now:

$ python3.6 -c 'import tokenize as t, io; 
print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n  y").readline)))'
b'if x:\n  y'

A quick search brought up multiple bug reports, among them

http://bugs.python.org/issue12691