tokenize.untokenize adding line continuation characters

[Rotwang]

On Tuesday, January 17, 2017 at 11:11:27 AM UTC, Peter Otten wrote:
> Rotwang wrote:
> 
> > Here's something odd I've found with the tokenize module: tokenizing 'if
> > x:\n    y' and then untokenizing the result adds '\\\n' to the end.
> > Attempting to tokenize the result again fails because of the backslash
> > continuation with nothing other than a newline after it. On the other
> > hand, if the original string ends with a newline then it works fine. Can
> > anyone explain why this happens?
> 
> > I'm using Python 3.4.3 on Windows 8. Copypasted from iPython:
> 
> This looks like a bug...
> 
> $ python3.4 -c 'import tokenize as t, io; 
> print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n  y").readline)))'
> b'if x:\n  y\\\n'
> 
> ...that is fixed now:
> 
> $ python3.6 -c 'import tokenize as t, io; 
> print(t.untokenize(t.tokenize(io.BytesIO(b"if x:\n  y").readline)))'
> b'if x:\n  y'
> 
> A quick search brought up multiple bug reports, among them
> 
> http://bugs.python.org/issue12691

Ah, thanks. I did search for bug reports before I started this thread but didn't find anything.