PEP 741 – Python Configuration C API
- Author:
- Victor Stinner <vstinner at python.org>
- Discussions-To:
- Discourse thread
- Status:
- Draft
- Type:
- Standards Track
- Created:
- 18-Jan-2024
- Python-Version:
- 3.13
- Post-History:
- 19-Jan-2024, 08-Feb-2024
Table of Contents
- Abstract
- Rationale
- Specification
- Examples
- Implementation
- Backwards Compatibility
- Rejected Ideas
- Discussions
- Copyright
Abstract
Add a C API to the limited C API to configure the Python initialization. It can be used with the stable ABI.
Complete PEP 587 API by adding PyInitConfig_AddModule()
which can be
used to add a built-in extension module; feature previously referred to
as the “inittab”.
Add PyConfig_Get()
and PyConfig_Set()
functions to
get and set the current runtime configuration at runtime.
PEP 587 “Python Initialization Configuration” unified all the ways to configure the Python initialization. This PEP (almost fully) unifies also the configuration of the Python preinitialization and the Python initialization in a single API, even if the preinitialization is still required to decode strings from the locale encoding.
This new API replaces the deprecated and incomplete legacy API which is scheduled for removal between Python 3.13 and Python 3.15.
Rationale
PyConfig is not part of the limited C API
When the first versions of PEP 587 “Python Initialization Configuration”
were discussed, there was a private field _config_version
(int
):
the configuration version, used for ABI compatibility. It was decided
that if an application embeds Python, it sticks to a Python version
anyway, and so there is no need to bother with the ABI compatibility.
The final PyConfig API of PEP 587 is excluded from the limited C API
since its main PyConfig
structure is not versioned. Python cannot
guarantee ABI backward and forward compatibility, it’s incompatible with
the stable ABI.
Since PyConfig was added to Python 3.8, the limited C API and the stable ABI are getting more popular. For example, Rust bindings such as the PyO3 project can target the limited C API to embed Python in Rust (but it’s not the default). In practice, PyO3 can use non-limited C API for specific needs, but using them avoids the stable ABI advantages.
Limitations of the legacy API
The legacy API to configure the Python initialization is based on the
legacy Py_Initialize()
function. It is now mostly deprecated:
- Set the initialization configuration such as
Py_SetPath()
: deprecated in Python 3.11 and removed in Python 3.13. - Global configuration variables such as
Py_VerboseFlag
: deprecated in Python 3.12 and scheduled for removal in Python 3.14. - Get the current configuration such as
Py_GetPath()
: deprecated in Python 3.13 and scheduled for removal in Python 3.15.
The legacy API doesn’t support the “Python Configuration” and the
“Isolated Configuration” of PEP 587 PyConfig API. It only provides a
“legacy configuration” of Py_Initialize()
which is in the between,
and also uses the legacy global configuration variables (such as
Py_VerboseFlag
).
Some options can only by set by environment variables, such as home
set by the PYTHONHOME
environment variable. The problem is that
environment variables are inherited by child processes which can be a
surprising and unwanted behavior.
Some configuration options, such as configure_locale
, simply cannot
be set.
Limitations of the limited C API
The limited C API is a subset of the legacy API. For example,
global configuration variables, such as Py_VerboseFlag
, are not
part of the limited C API.
While some functions were removed from the limited C API version 3.13, they are still part of the stable ABI. For example, building a application with the limited C API version 3.12 can still run with Python 3.13 stable ABI.
Get the runtime configuration
PEP 587 has no API to get the current runtime configuration, only to configure the Python initialization.
For example, the global configuration variable
Py_UnbufferedStdioFlag
was deprecated in Python 3.12 and using
PyConfig.buffered_stdio
is recommended instead. It only works to
configure Python, there is no public API to get
PyConfig.buffered_stdio
.
Users of the limited C API are asking for a public API to get the current runtime configuration.
Cython needs to get the optimization_level
configuration option:
issue.
When global configuration variables were deprecated in 2022, Marc-André Lemburg requested a C API to access these configuration variables at runtime (not only during Python initialization).
Security fix
To fix CVE-2020-10735,
a denial-of-service when converting very a large string to an integer (in base
10), it was discussed to add a new PyConfig
member to stable
branches which affects the ABI.
Gregory P. Smith proposed a different API using text based configuration
file to not be limited by PyConfig
members: FR: Allow private
runtime config to enable extending without breaking the PyConfig ABI
(August 2022).
In the end, it was decided to not add a new PyConfig
member to
stable branches, but only add a new PyConfig.int_max_str_digits
member to the development branch (which became Python 3.12). A dedicated
private global variable (unrelated to PyConfig
) is used in stable
branches.
Redundancy between PyPreConfig and PyConfig
The Python preinitialization uses the PyPreConfig
structure and the
Python initialization uses the PyConfig
structure. Both structures
have four duplicated members: dev_mode
, parse_argv
, isolated
and use_environment
.
The redundancy is caused by the fact that the two structures are
separated, whereas some PyConfig
members are needed by the
preinitialization.
Embedding Python
Applications embedding Python
Examples:
- Blender 3D graphics.
- fontforge font editor.
- Gimp.
- LibreOffice.
- OBS Studio.
- Tiled.
- vim text editor.
On Linux, FreeBSD and macOS, applications are usually either statically
linked to a libpython
, or load dynamically a libpython
. The
libpython
shared library is versioned, example:
libpython3.12.so
for Python 3.12 on Linux.
The vim project can target the stable ABI. Usually, the “system Python” version is used. It’s not currently possible to select which Python version to use. Users would like the ability to select a newer Python on demand.
On Linux, another approach to deploy an application embedding Python, such as GIMP, is to include Python in Flatpack, AppImage or Snap “container”. In this case, the application brings its own copy of Python version with the container.
Libraries embedding Python
Examples:
- Apache mod_wsgi (source).
- nimpy: Nim - Python bridge.
- PyO3: Rust bindings for the Python interpreter.
Utilities creating standalone applications
- py2app for macOS.
- py2exe for Windows.
- pyinstaller.
- PyOxidizer: it uses the PEP 587 PyConfig API.
These utilities create standalone applications, they are not linked to libpython.
Usage of a stable ABI
For tools like py2app/py2exe/pyinstaller, it is pretty inconvenient to have to rebuild the launcher executable that’s used to start the packaged application when there’s a bug fix release of Python.
You can’t extend a struct and assume embedding people all rebuild. They don’t. Real world embedding uses exist that use an installed Python minor version as a shared library. Update that to use a different sized struct in a public API and someone is going to have a bad time. That’s why I consider the struct frozen at rc1 time, even when only for use in the embedding / writing their own launcher case.
I am trying to embed the Python interpreter using a non C language. I have to stick with the limited API and private structures for configuration in headers files is a no-no. Basically, I need to be able to allocate and configure everything using only exportable functions and the heap… no private structure details.(…)
I am strictly limited to what’s in the shared library (DLL). I don’t have headers, I can’t statically “recompile” every time a new version of python comes out. That’s unmaintainable for me.
Quotes of Milian Wolff’s message:
Our application is a large complex C++ code base with lots of dependencies targeting all three major desktop platforms.Originally, we hoped to be able to use the stable python ABI to allow biologists to “bring your own python”. The idea was that they probably have a custom set of python libraries and code that they would like to continue using. Our integrated API - so we thought - was a tiny addition that should work with any Python out there, so we used the stable ABI.
This turned out to be a dead end, and I believe we can (should?) now use the non-stable ABI of python. Allowing end users to BYO Python caused far too much setup problems and support issues for us that it was not worth it in the end. Instead, we now rather want to ship a custom Python with a custom prefix that they can pip install custom libraries into as needed.
The problems we faced are not directly related to the stable ABI - quite the contrary. Rather, it was due to thirdparty python libraries that we shipped which themselves are not compatible across python version increments. E.g. for the integrated console we use qtconsole/jupyter, which worked in an archaic version with python 3.9 but requires newer versions for python 3.11+.
The ton of dependencies pulled in by UMAP was even worse, with numba and pydnndescent and llvmlite often taking months to support newer Python versions.
David Hewitt of the PyO3 project:
I think making the configuration structure opaque and using an API to set/get configuration by name is a welcome simplification:
- It’s a smaller API for language bindings like PyO3 to wrap and re-expose, and
- It’s easier for people to support multiple Python versions to embed into their application; no need to conditionally compile structure field access, can just use normal error handling if configuration values are not available for a specific version at runtime.
Quotes of Paul P. message:
I cannot agree more, it is the same story everywhere/every time CPython must be embedded. I maintened a runtime+ecosystem for Android 4.4+ for some time (in order more comfortably use Panda3D standalone than with Kivy), patching CPython and making a CI for it was ok.But I had to give up, because I had often to recompile every known modules: this is not sustainable for one individual.
So I dropped the Android arch to only go WebAssembly (Emscripten). But same (hard and boring) problem as always: have to rebuild numerous packages that are commonly used with 2D/3D framework. (…)
Except for ONE, Harfang3d. I did not rebuild this one since Python 3.11 initial port… Guess why? it is a limited C API - abi3 module!
Limited API abi3 are fresh air, fast and portable. And associated with a stable config runtime, it would be just perfect way!
See also issue gh-116139 building an
application embedding Python 3.11 and attempting to run it with Python
3.10: it does crash because the PyConfig
structure ABI is not stable
between two Python 3.x minor releases.
Set the runtime configuration
Marc-André Lemburg requested a C API to set the value of some configuration options at runtime:
optimization_level
verbose
parser_debug
inspect
write_bytecode
Previously, it was possible to set directly global configuration variables:
Py_OptimizeFlag
Py_VerboseFlag
Py_DebugFlag
Py_InspectFlag
Py_DontWriteBytecodeFlag
But these configuration flags were deprecated in Python 3.12 and are scheduled for removal in Python 3.14.
Specification
Add C API functions and structure to configure the Python initialization:
- Create config:
PyInitConfig
opaque structure.PyInitConfig_CreatePython()
.PyInitConfig_CreateIsolated()
.PyInitConfig_Free(config)
.
- Get options:
PyInitConfig_HasOption(config, name)
.PyInitConfig_GetInt(config, name, &value)
.PyInitConfig_GetStr(config, name, &value)
.PyInitConfig_GetWStr(config, name, &value)
.PyInitConfig_GetStrList(config, name, &length, &items)
.PyInitConfig_FreeStrList()
.PyInitConfig_GetWStrList(config, name, &length, &items)
.PyInitConfig_FreeWStrList()
.
- Set options:
PyInitConfig_SetInt(config, name, value)
.PyInitConfig_SetStr(config, name, value)
.PyInitConfig_SetStrLocale(config, name, value)
.PyInitConfig_SetWStr(config, name, value)
.PyInitConfig_SetStrList(config, name, length, items)
.PyInitConfig_SetStrLocaleList(config, name, length, items)
.PyInitConfig_SetWStrList(config, name, length, items)
.PyInitConfig_AddModule(config, name, initfunc)
- Initialize:
Py_PreInitializeFromInitConfig(config)
.Py_InitializeFromInitConfig(config)
.
- Error handling:
PyInitConfig_GetError(config, &err_msg)
.PyInitConfig_GetExitcode(config, &exitcode)
.
Add C API functions to get and set the current runtime configuration:
PyConfig_Get(name)
→object
.PyConfig_GetInt(name, &value)
.PyConfig_Set(name)
.PyConfig_Names()
→frozenset
.
The C API uses null-terminated UTF-8 encoded strings to refer to a configuration option.
All C API functions are added to the limited C API version 3.13.
The PyInitConfig
structure is implemented by combining the four
structures of the PyConfig
API and has an inittab
member as
well:
PyPreConfig preconfig
PyConfig config
PyStatus status
struct _inittab *inittab
forPyInitConfig_AddModule()
The PyStatus
status is no longer separated, but part of the unified
PyInitConfig
structure, which makes the API easier to use.
Configuration Options
Configuration options are named after PyPreConfig
and PyConfig
structure members. See the PyPreConfig documentation and
the PyConfig documentation.
Deprecating and removing configuration options is out of the scope of the PEP and should be discussed on a case by case basis.
Public configuration options
Following options can be get by PyConfig_Get()
and set and
PyConfig_Set()
.
Option | Type | Comment |
---|---|---|
argv |
list[str] |
API: sys.argv . |
base_exec_prefix |
str |
API: sys.base_exec_prefix . |
base_executable |
str |
API: sys.base_executable . |
base_prefix |
str |
API: sys.base_prefix . |
bytes_warning |
int |
API: sys.flags.bytes_warning . |
exec_prefix |
str |
API: sys.base_prefix . |
executable |
str |
API: sys.executable . |
inspect |
bool |
API: sys.flags.inspect (int ). |
int_max_str_digits |
int |
API: sys.flags.int_max_str_digits ,
sys.get_int_max_str_digits() and
sys.set_int_max_str_digits() . |
interactive |
bool |
API: sys.flags.interactive . |
module_search_paths |
list[str] |
API: sys.path . |
optimization_level |
int |
API: sys.flags.optimize . |
parser_debug |
bool |
API: sys.flags.debug (int ). |
platlibdir |
str |
API: sys.platlibdir . |
prefix |
str |
API: sys.base_prefix . |
pycache_prefix |
str |
API: sys.pycache_prefix . |
quiet |
bool |
API: sys.flags.quiet (int ). |
stdlib_dir |
str |
API: sys._stdlib_dir . |
use_environment |
bool |
API: sys.flags.ignore_environment (int ). |
verbose |
int |
API: sys.flags.verbose . |
warnoptions |
list[str] |
API: sys.warnoptions . |
write_bytecode |
bool |
API: sys.flags.dont_write_bytecode (int ) and sys.dont_write_bytecode (bool ). |
xoptions |
dict[str, str] |
API: sys._xoptions . |
Some option names are different than sys
attributes, such as
optimization_level
option and sys.flags.optimize
attribute.
PyConfig_Set()
sets the corresponding sys
attribute.
The xoptions
is a list of strings in PyInitConfig
where each
string has the format key
(value is True
implicitly) or
key=value
. In the current runtime configuration, it becomes a
dictionary (key: str
→ value: str | True
).
Read-only configuration options
Following options can be get PyConfig_Get()
, but cannot be set by
PyConfig_Set()
.
Option | Type | Comment |
---|---|---|
allocator |
int |
|
buffered_stdio |
bool |
|
check_hash_pycs_mode |
str |
API: imp.check_hash_pycs_mode . |
code_debug_ranges |
bool |
|
coerce_c_locale |
bool |
|
coerce_c_locale_warn |
bool |
|
configure_c_stdio |
bool |
|
configure_locale |
bool |
|
cpu_count |
int |
API: os.cpu_count() (int | None ). |
dev_mode |
bool |
API: sys.flags.dev_mode . |
dump_refs |
bool |
|
dump_refs_file |
str |
|
faulthandler |
bool |
API: faulthandler.is_enabled() . |
filesystem_encoding |
str |
API: sys.getfilesystemencoding() . |
filesystem_errors |
str |
API: sys.getfilesystemencodeerrors() . |
hash_seed |
int |
|
home |
str |
|
import_time |
bool |
|
install_signal_handlers |
bool |
|
isolated |
bool |
API: sys.flags.isolated (int ). |
legacy_windows_fs_encoding |
bool |
|
legacy_windows_stdio |
bool |
Windows only |
malloc_stats |
bool |
|
module_search_paths_set |
bool |
|
orig_argv |
list[str] |
API: sys.orig_argv . |
pathconfig_warnings |
bool |
|
parse_argv |
bool |
|
perf_profiling |
bool |
API: sys.is_stack_trampoline_active() . |
program_name |
str |
|
pythonpath_env |
str |
|
run_command |
str |
|
run_filename |
str |
|
run_module |
str |
|
run_presite |
str |
need a debug build. |
safe_path |
bool |
|
show_ref_count |
bool |
|
site_import |
bool |
API: sys.flags.no_site (int ). |
skip_source_first_line |
bool |
|
stdio_encoding |
str |
API: sys.stdin.encoding , sys.stdout.encoding and
sys.stderr.encoding . |
stdio_errors |
str |
API: sys.stdin.errors , sys.stdout.errors and
sys.stderr.errors . |
sys_path_0 |
str |
|
tracemalloc |
int |
API: tracemalloc.is_tracing() (bool ). |
use_frozen_modules |
bool |
|
use_hash_seed |
bool |
|
utf8_mode |
bool |
|
user_site_directory |
bool |
API: sys.flags.no_user_site (int ). |
warn_default_encoding |
bool |
|
_install_importlib |
bool |
|
_init_main |
bool |
|
_is_python_build |
bool |
|
_pystats |
bool |
API: sys._stats_on() , sys._stats_off() .
Need a Py_STATS build. |
Preinitialization
Calling Py_PreInitializeFromInitConfig()
preinitializes Python. For
example, it sets the memory allocator, and can configure the
LC_CTYPE
locale and configure the standard C streams such as
stdin
and stdout
.
The following options can only be set during the Python preinitialization:
allocator
,coerce_c_locale
,coerce_c_locale_warn
,configure_locale
,legacy_windows_fs_encoding
,utf8_mode
.
Trying to set these options after Python preinitialization fails with an error.
PyInitConfig_SetStrLocale()
and PyInitConfig_SetStrLocaleList()
functions call Py_PreInitializeFromInitConfig()
if Python is not
already preinitialized.
Create Config
PyInitConfig
structure:- Opaque structure to configure the Python preinitialization and the Python initialization.
PyInitConfig* PyInitConfig_CreatePython(void)
:- Create a new initialization configuration using default values
of the Python Configuration.
It must be freed with
PyInitConfig_Free()
.Return
NULL
on memory allocation failure. PyInitConfig* PyInitConfig_CreateIsolated(void)
:- Similar to
PyInitConfig_CreatePython()
, but use default values of the Isolated Configuration. void PyInitConfig_Free(PyInitConfig *config)
:- Free memory of an initialization configuration.
Get Options
The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string.
int PyInitConfig_HasOption(PyInitConfig *config, const char *name)
:- Test if the configuration has an option called name.
Return
1
if the option exists, or return0
otherwise. int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value)
:- Get an integer configuration option.
- Set *value, and return
0
on success. - Set an error in config and return
-1
on error.
- Set *value, and return
int PyInitConfig_GetStr(PyInitConfig *config, const char *name, char **value)
:- Get a string configuration option as a null-terminated UTF-8
encoded string.
- Set *value, and return
0
on success. - Set an error in config and return
-1
on error.
On success, the string must be released with
free(value)
. - Set *value, and return
int PyInitConfig_GetWStr(PyInitConfig *config, const char *name, wchar_t **value)
:- Get a string configuration option as a null-terminated wide string.
- Set *value and return
0
on success. - Set an error in config and return
-1
on error.
On success, the string must be released with
free(value)
. - Set *value and return
int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items)
:- Get a string list configuration option as an array of
null-terminated UTF-8 encoded strings.
- Set *length and *value, and return
0
on success. - Set an error in config and return
-1
on error.
On success, the string list must be released with
PyInitConfig_FreeStrList(length, items)
. - Set *length and *value, and return
void PyInitConfig_FreeStrList(size_t length, char **items)
:- Free memory of a string list created by
PyInitConfig_GetStrList()
. int PyInitConfig_GetWStrList(PyInitConfig *config, const char *name, size_t *length, wchar_t ***items)
:- Get a string list configuration option as an array of
null-terminated wide strings.
- Set *length and *value, and return
0
on success. - Set an error in config and return
-1
on error.
On success, the string list must be released with
PyInitConfig_FreeWStrList(length, items)
. - Set *length and *value, and return
void PyInitConfig_FreeWStrList(size_t length, wchar_t **items)
:- Free memory of a string list created by
PyInitConfig_GetWStrList()
.
Set Options
The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string.
Some configuration options have side effects on other options. This
logic is only implemented when Py_InitializeFromInitConfig()
is
called, not by the “Set” functions below. For example, setting
dev_mode
to 1
does not set faulthandler
to 1
.
int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value)
:- Set an integer configuration option.
- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int PyInitConfig_SetStr(PyInitConfig *config, const char *name, const char *value)
:- Set a string configuration option from a null-terminated UTF-8
encoded string. The string is copied.
- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int PyInitConfig_SetStrLocale(PyInitConfig *config, const char *name, const char *value)
:- Set a string configuration option from a null-terminated bytes
string encoded in the locale encoding. The string is copied.
The bytes string is decoded by
Py_DecodeLocale()
.Py_PreInitializeFromInitConfig()
must be called before calling this function.- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int PyInitConfig_SetWStr(PyInitConfig *config, const char *name, const wchar_t *value)
:- Set a string configuration option from a null-terminated wide
string. The string is copied.
- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int PyInitConfig_SetStrList(PyInitConfig *config, const char *name, size_t length, char * const *items)
:- Set a string list configuration option from an array of
null-terminated UTF-8 encoded strings. The string list is copied.
- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int PyInitConfig_SetStrLocaleList(PyInitConfig *config, const char *name, size_t length, char * const *items)
:- Set a string list configuration option from an array of
null-terminated bytes strings encoded in the locale encoding.
The string list is copied.
The bytes string is decoded by
Py_DecodeLocale()
.Py_PreInitializeFromInitConfig()
must be called before calling this function.- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int PyInitConfig_SetWStrList(PyInitConfig *config, const char *name, size_t length, wchar_t * const *items)
:- Set a string list configuration option from an error of
null-terminated wide strings. The string list is copied.
- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int PyInitConfig_AddModule(PyInitConfig *config, const char *name, PyObject* (*initfunc)(void))
:- Add a built-in extension module to the table of built-in modules.
The new module can be imported by the name name, and uses the function initfunc as the initialization function called on the first attempted import.
- Return
0
on success. - Set an error in config and return
-1
on error.
If Python is initialized multiple times,
PyInitConfig_AddModule()
must be called at each Python initialization.Similar to the
PyImport_AppendInittab()
function. - Return
Initialize Python
int Py_PreInitializeFromInitConfig(PyInitConfig *config)
:- Preinitialize Python from the initialization configuration.
- Return
0
on success. - Set an error in config and return
-1
on error.
- Return
int Py_InitializeFromInitConfig(PyInitConfig *config)
:- Initialize Python from the initialization configuration.
- Return
0
on success. - Set an error in config and return
-1
on error. - Set an exit code in config and return
-1
if Python wants to exit.
See
PyInitConfig_GetExitcode()
for the exitcode case. - Return
Error Handling
int PyInitConfig_GetError(PyInitConfig* config, const char **err_msg)
:- Get the config error message.
- Set *err_msg and return
1
if an error is set. - Set *err_msg to
NULL
and return0
otherwise.
An error message is an UTF-8 encoded string.
If config has an exit code, format the exit code as an error message.
The error message remains valid until another
PyInitConfig
function is called with config. The caller doesn’t have to free the error message. - Set *err_msg and return
int PyInitConfig_GetExitcode(PyInitConfig* config, int *exitcode)
:- Get the config exit code.
- Set *exitcode and return
1
if Python wants to exit. - Return
0
if config has no exit code set.
Only the
Py_InitializeFromInitConfig()
function can set an exit code if theparse_argv
option is non-zero. For example, an isolated configuration cannot set an exit code by default, sinceparse_argv
is zero by default.An exit code can be set when parsing the command line failed (exit code 2) or when a command line option asks to display the command line help (exit code 0).
- Set *exitcode and return
Get and Set the Runtime Configuration
The configuration option name parameter must be a non-NULL null-terminated UTF-8 encoded string.
PyObject* PyConfig_Get(const char *name)
:- Get the current runtime value of a configuration option as an
object.
- Return a new reference on success.
- Set an exception and return
NULL
on error.
The object type depends on the option: see Configuration Options tables.
Other options are get from internal
PyPreConfig
andPyConfig
structures.The caller must hold the GIL. The function cannot be called before Python initialization nor after Python finalization.
int PyConfig_GetInt(const char *name, int *value)
:- Similar to
PyConfig_Get()
, but get the value as an integer.- Set
*value
and return0
success. - Set an exception and return
-1
on error.
- Set
PyObject* PyConfig_Names(void)
:- Get all configuration option names as a
frozenset
.Set an exception and return
NULL
on error.The caller must hold the GIL.
PyObject* PyConfig_Set(const char *name, PyObject *value)
:- Set the current runtime value of a configuration option.
- Raise a
ValueError
if there is no option name. - Raise a
ValueError
if value is an invalid value. - Raise a
ValueError
if the option is read-only: cannot be set. - Raise a
TypeError
if value has not the proper type.
Read-only configuration options cannot be set.
The caller must hold the GIL. The function cannot be called before Python initialization nor after Python finalization.
- Raise a
Scope of the stable ABI
The limited C API and the stable ABI added by this PEP only provide a stable interface to program the Python initialization.
The behavior of options, the default option values, and the Python behavior can change at each Python version: they are not “stable”.
Moreover, configuration options can be added, deprecated and removed following the usual PEP 387 deprecation process.
Examples
Initialize Python
Example initializing Python, set configuration options of different types, return -1 on error:
int init_python(void)
{
PyInitConfig *config = PyInitConfig_CreatePython();
if (config == NULL) {
printf("PYTHON INIT ERROR: memory allocation failed\n");
return -1;
}
// Set an integer (dev mode)
if (PyInitConfig_SetInt(config, "dev_mode", 1) < 0) {
goto error;
}
// Set a list of wide strings (argv)
wchar_t *argv[] = {L"my_program", L"-c", L"pass"};
if (PyInitConfig_SetWStrList(config, "argv",
Py_ARRAY_LENGTH(argv), argv) < 0) {
goto error;
}
// Set a wide string (program name)
if (PyInitConfig_SetWStr(config, "program_name", L"my_program") < 0) {
goto error;
}
// Set a list of bytes strings (xoptions).
// Preinitialize implicitly Python to decode the bytes string.
char* xoptions[] = {"faulthandler"};
if (PyInitConfig_SetStrList(config, "xoptions",
Py_ARRAY_LENGTH(xoptions), xoptions) < 0) {
goto error;
}
// Initialize Python with the configuration
if (Py_InitializeFromInitConfig(config) < 0) {
goto error;
}
PyInitConfig_Free(config);
return 0;
error:
// Display the error message
const char *err_msg;
(void)PyInitConfig_GetError(config, &err_msg);
printf("PYTHON INIT ERROR: %s\n", err_msg);
PyInitConfig_Free(config);
return -1;
}
Increase initialization bytes_warning option
Example increasing the bytes_warning
option of an initialization
configuration:
int config_bytes_warning(PyInitConfig *config)
{
int bytes_warning;
if (PyInitConfig_GetInt(config, "bytes_warning", &bytes_warning)) {
return -1;
}
bytes_warning += 1;
if (PyInitConfig_SetInt(config, "bytes_warning", bytes_warning)) {
return -1;
}
return 0;
}
Get the runtime verbose option
Example getting the current runtime value of the configuration option
verbose
:
int get_verbose(void)
{
int verbose;
if (PyConfig_GetInt("verbose", &verbose) < 0) {
// Silently ignore the error
PyErr_Clear();
return -1;
}
return verbose;
}
On error, the function silently ignores the error and returns -1
. In
practice, getting the verbose
option cannot fail, unless a future
Python version removes the option.
Implementation
Backwards Compatibility
Changes are fully backward compatible. Only new APIs are added.
Existing API such as the PyConfig
C API (PEP 587) are left
unchanged.
Rejected Ideas
Configuration as text
It was proposed to provide the configuration as text to make the API compatible with the stable ABI and to allow custom options.
Example:
# integer
bytes_warning = 2
# string
filesystem_encoding = "utf8" # comment
# list of strings
argv = ['python', '-c', 'code']
The API would take the configuration as a string, not as a file. Example
with a hypothetical PyInit_SetConfig()
function:
void stable_abi_init_demo(int set_path)
{
PyInit_SetConfig(
"isolated = 1\n"
"argv = ['python', '-c', 'code']\n"
"filesystem_encoding = 'utf-8'\n"
);
if (set_path) {
PyInit_SetConfig("pythonpath = '/my/path'");
}
}
The example ignores error handling to make it easier to read.
The problem is that generating such configuration text requires adding quotes to strings and to escape quotes in strings. Formatting an array of strings becomes non-trivial.
Providing an API to format a string or an array of strings is not really worth it, whereas Python can provide directly an API to set a configuration option where the value is passed directly as a string or an array of strings. It avoids giving special meaning to some characters, such as newline characters, which would have to be escaped.
Refer to an option with an integer
Using strings to refer to a configuration option requires comparing strings which can be slower than comparing integers.
Use integers, similar to type “slots” such as Py_tp_doc
, to refer to
a configuration option. The const char *name
parameter is replaced
with int option
.
Accepting custom options is more likely to cause conflicts when using integers, since it’s harder to maintain “namespaces” (ranges) for integer options. Using strings, a simple prefix with a colon separator can be used.
Integers also requires maintaining a list of integer constants and so make the C API and the Python API larger.
Python 3.13 only has around 62 configuration options, and so performance is not really a blocker issue. If better performance is needed later, a hash table can be used to get an option by its name.
If getting a configuration option is used in hot code, the value can be read once and cached. By the way, most configuration options cannot be changed at runtime.
Fully remove the preinitialization
Delay decoding
Without PyInitConfig_Get*()
functions, it would be possible to store
PyInitConfig_SetStrLocale()
and PyInitConfig_SetStrLocaleList()
strings encoded and only initialize the LC_CTYPE
locale and
decode the strings in Py_InitializeFromInitConfig()
.
The problem is that users asked for PyInitConfig_Get*()
functions.
For example, PyInitConfig_GetStr()
must decode the string from the
locale encoding and then encode it to the UTF-8 encoding.
However, if PyInitConfig_SetStrLocale()
and
PyInitConfig_SetStrLocaleList()
strings are decoded as designed by
the PEP, there is no risk of mojibake: PyInitConfig_GetStr()
returns
the expected decoded strings.
Remove the Python configuration
If PyInitConfig_CreatePython()
is removed, the preinitialization is
no longer needed since the LC_CTYPE
is not configured by default by
PyInitConfig_CreateIsolated()
and setting "configure_locale"
option can always fail.
The problem is that users asked to be able to write their own customized
Python, so have a Python-like program but with a different default
configuration. The PyInitConfig_CreatePython()
function is needed
for that.
Moreover, the Python configuration is also part of the PEP 587 design, implemented in Python 3.8.
Disallow setting the "configure_locale"
option has similar issues.
Multi-phase initialization (similar to PEP 432)
Eric Snow expressed concerns that this proposal might reinforce with embedders the idea that initialization is a single monolithic step. He argued that initialization involves 5 distinct phases and even suggested that the API should reflect this explicitly. Eric proposed that, at the very least, the implementation of initialization should reflect the phases, in part for improved code health. Overall, his explanation has some similarities with PEP 432 and PEP 587.
Another of Eric’s key points relevant to this PEP was that, ideally,
the config passed to Py_InitializeFromConfig()
should be complete
before that function is called, whereas currently initialization
actually modifies the config.
While Eric wasn’t necessarily suggesting an alternative to PEP 741,
any proposal to add a granular initialization API around phases is
effectively the opposite of what this PEP is trying to accomplish.
Such API is more complicated, it requires adding new public structures
and new public functions. It makes the Python initialization more
complicated, rather than this PEP tries to unifiy existing APIs and make
them simpler (the opposite). Having multiple structures for similar
purpose can lead to duplicate members, similar issue than duplicated
members between existing PyPreConfig
and PyConfig
structures.
Discussions
- PEP 741: Python Configuration C API (second version) (February 2024).
- PEP 741: Python Configuration C API (January 2024).
- FR: Allow private runtime config to enable extending without breaking the PyConfig ABI (August 2022).
Copyright
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
Source: https://github.com/python/peps/blob/main/peps/pep-0741.rst
Last modified: 2024-03-18 13:33:22 GMT