[Python-Dev] UTF-8 Mode now also enabled by the POSIX locale

[Victor Stinner]

Hi,

While working on test_utf8_mode on AIX (bpo-34347) and HP-UX
(bpo-34403), I noticed that FreeBSD doesn't work properly with the
POSIX locale (bpo-34527). I also noticed that my implementation of my
PEP 540 "UTF-8 Mode" doesn't respect the PEP: the UTF-8 Mode should be
enabled by the POSIX locale, not only by the C locale.

I just modified Python 3.7 and master (future 3.8) to enable UTF-8
Mode if the LC_CTYPE locale is "POSIX":
https://bugs.python.org/issue34527

I also fixed FreeBSD to support the "POSIX" locale as well (3.6, 3.7
and master branches).

Note: The C locale coercion (PEP 538) is only enabled if the LC_CTYPE
locale is "C".

To finish my list of recent Unicode issues, I'm also working on
bpo-34523: select the Python filesystem encoding before
Py_Initialize(). This change will allow embedders to force an encoding
instead of letting Python choose one for them.

Victor