[Python-Dev] Moving forward on the object memory API

[Tim Peters]

[Tim]
>> Is there a reason not to make these the simpler
>>
>> #define PyMem_MALLOC malloc

[Neil Schemenauer]
> It adds a type cast in some cases.  We should keep is simple were it
> doesn't matter.

Is it ever necesssary to add a type cast?  For example, under ANSI C, free()
is declared to take a void* argument, and C will automatically cast whatever
is passed to it to void*.  Adding our own (void*) cast is redundant.  We can
assume all extensions include Python.h, which in turn includes stdlib.h, so
the proper system prototypes should always be in scope in extensions.

A difficulty with doing more than plain name substitution is that, e.g.,

    void *p = (void*)&PyMem_FREE;

doesn't work right if PyMem_FREE doesn't expand to (just) a function name.
But so far the docs are silent on whether such uses are supported.

> PyMem_FREE would call free() by PyMem_Free would call _PyMalloc_Free.
> PyMem_FREE is relatively new.  1.5.2 only had PyMem_DEL and that was the
> recommended way to free object memory.  PyMem_Free is new too but based
> on my survey, some people use it for freeing object memory.

Ya, and I don't know how to resolve this.

[on PyMem_{xxx, XXX}]
> I guess we could deprecate the uppercase versions and make the lowercase
> ones macros.

The docs already deprecated the uppercase names.  Whether the lowercase
names can be macros depends on what they expand to (binary compatibility
again).

> As far as warnings go, I think the alternative spellings should be
> allowed for a long time to come.  We can't just keep changing
> our tune and expect the extension writers dance. :-)

I don't expect they'll ever go away, which is why I've got little enthusiasm
for arguing about "deprecation".

> ...
> I thought we decided that making PyMem_MALLOC do anything but malloc()
> was hopeless?  The only reason the PyMem_* layer is there is that so
> adventurous people can replace the system malloc() with something like a
> conservative GC malloc.  In that case the extensions would have to be
> recompiled.

Actually, PyMem_MALLOC probably *should* make the same guarantee as
PyMem_Malloc about what happens if you pass it 0.  And we can never be sure
that "an important" platform won't sprout a bug that we need to patch over
with a more-or-less trivial wrapper.  As Guido just covered in another msg,
PyMem_Malloc is also needed to ensure that an extension gets its memory from
the same malloc the Python DLL is using on Windows.  This can be important
if it's possible for the corresponding free() to be done *by* the Python
DLL.  Or so I've been told.

> I was just trying to cut down the number of functions and macros.

Which I appreciate!  Guido will want to do that too, I'm just warning that
after he thinks about it <wink>, he'll retract at least one of his +1s and
swing to the opposite pole.

> ...
> Extensions can use PyObject_{Malloc,Realloc,Free} but the type based
> allocators are preferred.

They presumably exist for extensions that want to use the same allocator as
the object allocator uses.  For example, someone who has tons of raw little
C memory blobs that aren't objects, but that could benefit from exploiting
the pymalloc malloc.

> Extensions should not use PyObject_{MALLOC,REALLOC,FREE}.

This would be consistent with what the docs say about PyMem_UPPERCASE.  I
think Guido likes to use lowercase/UPPERCASE to distinguish solely between
whether a thing is *implemented* via a function or a macro, and I think
that's where some of the historical confusion comes from.  As a user, I
don't care at all how a thing is implemented, and much prefer
UPPERCASE/lowercase to distinguish whether binary compatibility is
guaranteed.  And that view appears to be shared by people who changed the
memory API after Guido.  So what lowercase/UPPERCASE is *intended* to convey
has become another murky mess.  That's another argument that's yet to begin.

> I forget about PyObject_Resize and PyObject_GC_Resize.  They should be
> part of the preferred API.

Curiously, there is no PyObject_Resize:  PyObject_GC_Resize is our only
general object resize function.