[Python-Dev] Enhancement of Python memory allocators

[Victor Stinner]

Hi,

I would like to improve memory allocators of Python. My two use cases
are replacing memory allocators with custom allocators in embedded
system and hooking allocators to track usage of memory.

I wrote a patch for this, I'm going to commit it if nobody complains:
http://bugs.python.org/issue3329

Using this patch, detecting memory corruptions (buffer underflow and
overflow) can be done without recompilation. We may add an environment
variable to enable Python debug functions at runtime, example:
PYDEBUGMALLOC=1. There is just a restriction: the environment variable
would not be ignored with -E command line option, because command line
options are parsed after the first memory allocation. What do you
think?

*****

The patch adds the following functions:

void PyMem_GetAllocators(
    void **ctx_p,
    void* (**malloc_p) (void *ctx, size_t size),
    void* (**realloc_p) (void *ctx, void *ptr, size_t size),
    void (**free_p) (void *ctx, void *ptr));

void PyMem_SetAllocators(
    void *ctx,
    void* (*malloc) (void *ctx, size_t size),
    void* (*realloc) (void *ctx, void *ptr, size_t size),
    void (*free) (void *ctx, void *ptr));

It adds 4 similar functions (get/set) for PyObject_Malloc() and
allocators of pymalloc arenas.

*****

For the "track usage of memory" use case, see the following project
which hooks memory allocators using PyMem_SetAllocators() and
PyObject_SetAllocators() to get allocated bytes per filename and line
number.
https://pypi.python.org/pypi/pytracemalloc

*****

Another issue proposes to use VirtualAlloc() and VirtualFree() for
pymalloc arenas, see:
http://bugs.python.org/issue13483

I don't know if it would be interesting, but it would now possible to
choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc,
...) at runtime, with an environment variable for example.

Victor