[issue36202] Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

Calling Py_DecodeLocale() before Py_Initialize() or _Py_InitializeCore() is broken since Python 3.7 if the C locale coercion (PEP 538) or UTF-8 mode (PEP 540) changes the encoding in the middle of _Py_InitializeCore().

I added a new phase to the Python initialization in bpo-36142, a new _PyPreConfig structure, which can be used to fix this mojibake issue.

The code for embedding Python should look like:
---
_Py_PreInitialize();

_PyCoreConfig config;
config.home = Py_DecodeLocale("/path/to/home");

_PyInitError err = _Py_InitializeFromConfig(&config);
if (_Py_INIT_FAILED(err)) {
    _PyCoreConfig_Clear(&config);
    _Py_ExitInitError(err);
}

/* use Python here */

Py_Finalize();
_PyCoreConfig_Clear(&config);
---

Except that there is no _Py_PreInitialize() function yet :-)

----------
components: Interpreter Core
messages: 337252
nosy: ncoghlan, vstinner
priority: normal
severity: normal
status: open
title: Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36202>
_______________________________________