[issue10117] Tools/scripts/reindent.py fails on non-UTF-8 encodings

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

> Leaving open to discuss whether anything can/should be done
> for the case when reindent acts as an stdin

sys.stdin.buffer and sys.stdout.buffer should be used with tokenize.detect_encoding(). We may read first stdin and write it into a BytesIO object to be able to rewind after detect_encoding. Something like:

content = sys.stdin.buffer.read()
raw = io.BytesIO(content)
buffer = io.BufferedReader(raw)
encoding, _ = detect_encoding(buffer.readline)
buffer.seek(0)
text = TextIOWrapper(buffer, encoding)
# use text

----------
nosy: +haypo

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10117>
_______________________________________