[issue35265] Internal C API: pass the memory allocator in a context

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

Currently, the Python initialization code base is full of:

static void
pymain_clear_cmdline(_PyMain *pymain, _PyCmdline *cmdline)
{
    PyMemAllocatorEx old_alloc;
    _PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
    ...
    PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
}

It's a hack that I wrote when I worked on the Python initialization code, to make sure that we use the same memory allocator to allocate and release memory. The problem is that Python allocates memory at startup, then the memory allocator can be changed in Py_Initialize() (ex: by PYTHONMALLOC environment variable), but at exit, we need to reuse the same memory allocator that we used to allocate memory early.

Instead of "forcing" the memory allocator to "default allocator", I propose to add a new "context" structure which contains the memory allocator (PyMemAllocatorEx structure). Example:

typedef struct {
    /* Enable UTF-8 mode?
       Set by -X utf8 command line option and PYTHONUTF8 environment variable.
       If set to -1 (default), inherit Py_UTF8Mode value. */
    int utf8_mode;

    PyMemAllocatorEx raw_alloc;
} _PyConfigCtx;

The context should be passed to any function allocating directly or indirectly memory.

Py_Main(), "core config" and "path config" are good target for these changes. These functions are used during Python initialization.

I know that Eric Snow would like to pass a "context" to functions of the C API. I don't want to make a giant change of the C API. I only want to modify the internal C API, and only modifiy Python internal. The context that I propose is designed for early Python initialization, and fix the practical problem of memory allocator.

Later we can discussed how _PyRuntime and the pointer to the current interpreter can be added into this context, or if a new context should be created.

Attached PR is a proof-of-concept of my idea, but the giant PR might be splitted into smaller changes to more incremental changes.

----------
components: Interpreter Core
messages: 330001
nosy: eric.snow, ncoghlan, vstinner
priority: normal
severity: normal
status: open
title: Internal C API: pass the memory allocator in a context
type: enhancement
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35265>
_______________________________________