[Python-Dev] Activating pymalloc

[Tim Peters]

[Guido, on PyMem_XXX still calling malloc/free even when pymalloc is
 enabled]

> Well, then you would have to provide yet another set of macros that
> don't require the GIL, for use by extensions that need to allocate
> memory while not holding the GIL.  Of course, you could say, "then
> just use malloc/free", but the intention of pymem.h was that it would
> *also* be possible to provide a malloc/free substitute that *was*
> thread-safe -- unlike pymalloc.  I don't know how realistic that is --
> the only candidate would be GNU malloc but that would just replace the
> malloc/realloc/free entry points so wouldn't need any of the
> pymem.h-approved hackery.

That's an old sore point:  the memory API is, in its many "hidden" layers,
almost pure YAGNI.  Vladimir found it useful when he was exploring the
universe of all possible memory allocation algorithms, but I've seen no
evidence that anyone else cares (or even understands it).

Rather than have PyMem_XXX not call malloc/free, I expect what I'd like more
is a way for core code to say "use obmalloc.c here, even though I'm not
allocating 'an object' at this point".  This could be used for, e.g., list
guts and especially for Unicode string guts.  But despite all the layers of
macros here, there doesn't appear to be a blessed way to spell that!  The
malloc substitute in obmalloc.c is

_THIS_MALLOC(size_t nbytes)

and _THIS_MALLOC is a macro exapnding to PyCore_OBJECT_MALLOC_FUNC, and
PyCore_OBJECT_MALLOC_FUNC is a macro expanding to _PyCore_ObjectMalloc when
pymalloc is enabled.  _PyCore_ObjectMalloc is not a macro, it's the actual
ultimate name of the malloc substitute (ignoring the possibilities for
somone to break into the macro chain).  All *these* macro names are supposed
to be "invisible" (internal to the memory implementation).

In the other direction, e.g.,

   PyObject_New ->
   _PyObject_New ->
   PyObject_MALLOC ->
   PyCore_OBJECT_MALLOC ->
   PyCore_OBJECT_MALLOC_FUNC,
       and we're back in the macro chain traced above

There's just no sane way to say "give me obmalloc.c's malloc".  I'd rather
we named obmalloc.c's entry points, e.g., PyMalloc_Malloc directly ...