[issue23993] Use surrogateescape error handler by default in open() if the LC_CTYPE locale is C at startup

[Nick Coghlan]

Nick Coghlan added the comment:

If a Linux distro is using systemd (which is essentially all recent versions of popular distros, including RHEL/CentOS, although it won't land in Ubuntu LTS until 16.04), then cron jobs and service daemons will get their locale set properly based on the contents of /etc/locale.conf. Thus "use an init system that reliably sets the locale correctly for cron jobs and service daemons" is the correct fix for this problem.

Unfortunately, there are still an awful lot of Linux systems out there using other init systems that don't reliably set the locale, and for those "Python 3 shouldn't be worse than Python 2" is a desirable behavioural goal here.

Thus, I think it makes sense for Python to special case the C locale by assuming it's always the wrong setting, and thus surrogateescape is going to be needed on all system interfaces. While it won't be a perfect fix, at least we'll be able to roundtrip data within the system appropriately, even if it still gets corrupted in the face of encoding conversions.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue23993>
_______________________________________