[New-bugs-announce] [issue42260] [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

This issue is a follow-up of the PEP 567 which introduced the PyConfig C API and is related to PEP 432 which wants to rewrite Modules/getpath.c in Python.

I would like to add a new PyInterpreterState_SetConfig() function to be able to reconfigure a Python interpreter in C. One example is to write a custom sys.path, to implement of virtual environment (common request for embedded Python), etc. Currently, it's really complex to tune the Python configuration.

The use case is to tune Python for embedded Python. First, I would like to add new functions to the C API for that:

* PyInterpreterState_GetConfigCopy()
* PyInterpreterState_SetConfig()

The second step will to be expose these two functions in Python (I'm not sure where for now), and gives the ablity to tune the Python configuration in pure Python.

The site module already does that for sys.path, but it is running "too late" in the Python initialization. Here the idea is to configure Python before it does access any file on disk, after the "core" initialization and before the "main" initialization.

One concrete example would be to reimplement Modules/getpath.c in Python, convert it to a frozen module, and run it at Python startup to populate sys.path. It would allow to move some of the site code into this module to run it earlier.

Pseudo-code in C:
---------------------
void init_core(void)
{
  // "Core" initialization
  PyConfig config;
  PyConfig_InitPython(&config);
  PyConfig._init_main = 0
  Py_InitializeFromc(&config);
  PyConfig_Clear(&config);
}

void tune_config(void)
{
  PyConfig config;
  PyConfig_InitPython(&config);

  // Get a copy of the current configuration
  PyInterpreterState_GetConfigCopy(&config);  // <== NEW API!

  // ... put your code to tune config ...

  // dummy example, current not possible in Python
  config.bytes_warnings = 1;

  // Reconfigure Python with the updated configuration
  PyInterpreterState_SetConfig(&config);  // <=== NEW API!
  PyConfig_Clear(&config);
}
  
int main()
{
  init_core();
  tune_config(); // <=== THE USE CASE!
  _Py_InitializeMain();
  return Py_RunMain();
}
---------------------

In this example, tune_config() is implemented in C. But later, it will be possible to convert the configuration to a Python dict and run Python code to tune the configuration.

The PEP 587 added a "Multi-Phase Initialization Private Provisional API":

* PyConfig._init_main = 0
* _Py_InitializeMain()

https://docs.python.org/dev/c-api/init_config.html#multi-phase-initialization-private-provisional-api

----------
components: C API
messages: 380327
nosy: vstinner
priority: normal
severity: normal
status: open
title: [C API] Add PyInterpreterState_SetConfig(): reconfigure an interpreter
versions: Python 3.10

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42260>
_______________________________________