[Python-Dev] Moving forward on the object memory API

[Neil Schemenauer]

Tim Peters wrote:
> Is there a reason not to make these the simpler
> 
> #define PyMem_MALLOC malloc
> 
> etc?

It adds a type cast in some cases.  We should keep is simple were it
doesn't matter.

> > I think making PyMem_FREE call free() is safe.  I haven't seen any
> > extension modules allocate with PyObject_NEW and free with PyMem_FREE.
> 
> What about PyMem_Free()?  You reported that MySQL-python-0.3.5 and crng-1.1
> mix PyObject_NEW with PyMem_Free.

PyMem_FREE would call free() by PyMem_Free would call _PyMalloc_Free.
PyMem_FREE is relatively new.  1.5.2 only had PyMem_DEL and that was the
recommended way to free object memory.  PyMem_Free is new too but based
on my survey, some people use it for freeing object memory.


> > We deprecate the PyMem_* functions.
> 
> This may hit two snags:  the docs *currently* say (and have for some time)
> that all of
> 
>    PyMem_{MALLOC, REALLOC, FREE, NEW, RESIZE, DEL}
> 
> are "deprecated in extension modules".  The docs encourage using the PyMem
> function spellings instead (or perhaps the docs insist on that -- it depends
> on what the docs mean by "deprecated"; IMO deprecation is without
> consequence in any context where we can't deliver warning msgs for at least
> a release cycle -- else we'll never actually get rid of "deprecated" stuff).

I guess we could deprecate the uppercase versions and make the lowercase
ones macros.  As far as warnings go, I think the alternative spellings
should be allowed for a long time to come.  We can't just keep changing
our tune and expect the extension writers dance. :-)

> The rationale has to do with cross-release binary compatibility:  if an
> extension spells a thing PyMem_Malloc, they have "a right" to expect that
> their extension won't need to be recompiled across releases just because
> Python changes what PyMem_MALLOC does.

I thought we decided that making PyMem_MALLOC do anything but malloc()
was hopeless?  The only reason the PyMem_* layer is there is that so
adventurous people can replace the system malloc() with something like a
conservative GC malloc.  In that case the extensions would have to be
recompiled.

> In any case, I predict Guido will say that the function spellings must
> indeed be functions, and are still the recommended way

That's fine.  I was just trying to cut down the number of functions and
macros.

> > PyObject_{New,NewVar,Del}, and PyObject_GC_{New,NewVar,Del}.
> 
> This part should be non-controversial so far as it goes.  But the docs don't
> currently give any cautions about using the macro versions of PyObject_XXX,
> and in addition to those we're also missing PyObject_{Malloc, MALLOC,
> Realloc, REALLOC, Free, FREE} and PyObject_GC_Resize in this account.
> Straightening all that out again requires agreeing on what the rules really
> are.  I think ultimately has to come from Guido, but the best way to get
> that to happen is to provoke him with schemes he doesn't like <wink>.

Extensions can use PyObject_{Malloc,Realloc,Free} but the type based
allocators are preferred.  Extensions should not use
PyObject_{MALLOC,REALLOC,FREE}.

I forget about PyObject_Resize and PyObject_GC_Resize.  They should be
part of the preferred API.

  Neil