[issue19846] Python 3 raises Unicode errors with the C locale

[STINNER Victor]

STINNER Victor added the comment:

> The fact that write() uses sys.getfilesystemencoding() is either a defect or a bad design (I leave the decision to you).

"Standard streams (sys.stdin, sys.stdout, sys.stderr) uses the locale encoding. sys.stdin and sys.stdout use the strict error handler, sys.stderr uses the backslashreplace error handler. These encodings and error handlers can be overriden by the PYTHONIOENCODING. Since Python 3.3, it's possible to only set the error handler using ":errors" syntax (ex: PYTHONIOENCODING=":replace")."

stdout uses the locale encoding (and if you read my whole message, you may understand why sys.getfilesystemencoding() is also the locale encoding on UNIX).

(FYI on Windows, the OEM code page is used for standard streams.)

sys.getdefaultencoding() is always utf-8, this is unrelated to standard streams and OS data: it's the default value of the encoding parameter of str.encode() and str.decode(). I'm surprised that it's not documented to be utf-8, it is hardcoded and so always utf-8 in Python 3.

> But I'm still missing a reply to my suggestion. As I'm seeing it has no disadvantages to give the developer optionally the control.

"Standard streams (sys.stdin, sys.stdout, sys.stderr) uses the locale encoding. sys.stdin and sys.stdout use the strict error handler, sys.stderr uses the backslashreplace error handler. These encodings and error handlers can be overriden by the PYTHONIOENCODING. Since Python 3.3, it's possible to only set the error handler using ":errors" syntax (ex: PYTHONIOENCODING=":replace")."

If the environment variable is not enough, see also #15216 which proposes to add a TextIOWrapper.set_encoding() method. (I'm not really a fan of this proposition, but it looks like some users ask for it.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19846>
_______________________________________