[Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

[Victor Stinner]

> * Add new GIL-free (no need to hold the GIL) memory allocator functions:
>
>   - ``void* PyMem_RawMalloc(size_t size)``
>   - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)``
>   - ``void PyMem_RawFree(void *ptr)``
>   - the behaviour of requesting zero bytes is not defined: return *NULL*
>     or a distinct non-*NULL* pointer depending on the platform.
> (...)
> * Add new functions to get and set internal functions of
>   ``PyMem_Malloc()``, ``PyMem_Realloc()`` and ``PyMem_Free()``:
>
>   - ``void PyMem_GetAllocator(PyMemBlockAllocator *allocator)``
>   - ``void PyMem_SetAllocator(PyMemBlockAllocator *allocator)``
>   - ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return
>     *NULL*: it would be treated as an error.
>   - default allocator: ``malloc()``, ``realloc()``, ``free()``;
>     ``PyMem_Malloc(0)`` calls ``malloc(1)``
>     and ``PyMem_Realloc(NULL, 0)`` calls ``realloc(NULL, 1)``

Oh, one more question: PyMem_RawMalloc(0) has an undefined behaviour,
whereas PyMem_Malloc(0) has a well defined behaviour (don't return
NULL). Adding "if (size == 1) size = 0;" in the default implementation
of PyMem_RawMalloc(0) should not have a visible overhead, but it gives
the same guarantee than PyMem_Malloc(0) (don't return NULL). Do you
agree to add the test?

I chose to implement "if (size > (size_t)PY_SSIZE_T_MAX) return NULL;"
in Py*_Malloc(), whereas "if (size == 0) size =1;" is implemented in
the inner function (_PyMem_Malloc). An application may use an
allocator which has already a well defined behaviour (a "malloc(0)"
that don't return NULL) and I expect malloc(1) to allocate "more"
memory than malloc(0) (malloc(0) may create a singleton) :-)

Victor