[New-bugs-announce] [issue5093] 2to3 with a pipe on non-ASCII script

[STINNER Victor]

New submission from STINNER Victor <victor.stinner at haypocalc.com>:

If Python output is redirected to a pipe, sys.stdout encoding is 
ASCII. So "2to3 script.py|cat" will write the patch in ASCII. If the 
script contains a non-ASCII character, 2to3 fails with:
  ...
  File ".../lib2to3/refactor.py", line 238, in refactor_file
    self.processed_file(str(tree)[:-1], filename, write=write)
  File ".../lib2to3/refactor.py", line 342, in processed_file
    self.print_output(diff_texts(old_text, new_text, filename))
  File ".../main.py", line 48, in print_output
    print(line)
UnicodeEncodeError: 'ascii' codec can't encode character '\xfb' in 
position 11: ordinal not in range(128)

Should we consider the input file and stdout as binary files? 
Workaround: modify the files in place (-w option) but don't write the 
patch to stdout (no such option yet).

A project may contain scripts in ASCII, Latin-1 and UTF-8 (eg. Python 
source code ;-)).

----------
components: 2to3 (2.x to 3.0 conversion tool)
messages: 80733
nosy: haypo
severity: normal
status: open
title: 2to3 with a pipe on non-ASCII script

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5093>
_______________________________________