[Python-ideas] PEP 538 (C locale coercion) now depends on PEP 540 (UTF-8 mode)

[Nick Coghlan]

Hi folks,

I've pushed an update to PEP 538 that eliminates any fallback to the
en_US.UTF-8 locale, and instead relies solely on PEP 540's bypassing
of the platform's locale subsystem in cases where the default C locale
is specified.

This revision means that the two PEPs can now be read as follows:

* PEP 540 will make CPython's default behaviour significantly more
user friendly when the OS claims that the system API encoding is
ASCII, at the cost of making it inconsistent with other C/C++
components in the same process
* PEP 538 is then a follow-on proposal to fix that inconsistency by
coercing the C/C++ locale settings to better align with CPython's
decision to ignore the nominal ASCII encoding

I won't repost the whole thing here, but will instead provide relevant
links and note a few particular highlights of the changes:

* Full PEP: https://www.python.org/dev/peps/pep-0538/
* Diff: https://github.com/python/peps/commit/481573aa2722f515b38e30beaaffc5e1fb9bbfb4

Major changes:

* en_US-UTF-8 fallback eliminated in favour of the PYTHONUTF8 feature in PEP 540
* "LC_CTYPE=UTF-8" fallback added with the aim of improving behaviour
on *BSD systems
* surrogateescape is enabled on standard streams by default (see
behavioural examples in the PEP for rationale)
* notes that the proposal makes it possible to fix an interaction bug
with the GNU readline module on Android

I haven't updated the reference implementation yet, as I'd prefer to
see PEP 540 resolved and merged before doing that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia