[Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation

[Victor Stinner]

> IMHO the implementation is really a secondary concern here, the main
> question is: what is the correct behavior?
>
> Nick:
>
> * Do we agree that we need to provide a way to disable C locale
> coercion (PEP 538) even when -E is used?
> * Do you agree that Py_Initialize() and Py_Main() must not enable the
> C locale coercion (PEP 538)?
>
> I understood that your reply is yes for the second question, since you
> insist to push your change which also prevent Py_Initialize() and
> Py_Main() to enable C locale coercion.

Hum, I'm not sure if I explained properly my opinion on these questions.

I consider that Python 3.7.0 introduced a regression compared to
Python 3.6: it changes the LC_CTYPE locale for Python and all child
processes and it's not possible to opt-out for that when using -E
command line option. I proposed (and implemented) -X coerce_c_locale=0
for that. Unicode and locales are so hard to get right that I consider
that it's important that we provide an option to opt-out,. Otherwise,
someone will find an use case where Python 3.7 doesn't behave as
expected and break one specific use case. I didn't notice a complain
yet, but there are very few Python 3.7 users at this point. For
example, very few Linux distributions use it yet.

I consider that PYTHONCOERCECLOCALE must not introduce an exception in
-E: it must be ignored when -E or -I is used. For security reasons,
it's important to really ignore all PYTHON* environment variables.
"Unicode" (in general) has been abused in the past to exploit
vulnerabilities in applications. Locales and encodings are so hard,
that it's easy to mess up and introduce a vulnerability just caused by
encodings. It's also important to get deterministic and reproducible
programs.

For Py_Initialize() and Py_Main(): I have no opinion, so I rely on
Nick's request to make sure that the C locale is not coerced when
Python is embeded :-)

Victor