[Python-Dev] Activating pymalloc

[Guido van Rossum]

> [Guido]
> > Some extensions will break because they don't use the right
> > alloc/dealloc macros (we just fixed a few of these in Zope)

[Tim]
> Are "the right" alloc/dealloc macros documented anywhere?  I can't
> find it, if they are.

There's a long comment block at the start of objimpl.h documenting
PyObject_New, PyObject_NewVar, PyObject_Del, and PyObject_Init.  These
are the only ones that 3rd party extensions should use for allocating
storage for *objects*.  In particular, a type's tp_dealloc function
should use PyObject_Del when the type's constructor uses PyObject_New
or a variant.  It ends with:

    Unless you have specific memory management requirements, it is
    recommended to use PyObject_{New, NewVar, Del}. */

While that's pretty hidden, it's also pretty unambiguous on what "the
right" thing is.

Unfortunately, *core* object types are allowed to use other macros as
well, and many 3rd party extensions copy examples from the core rather
than reading comments in obscure header files.

There's also the question of how to allocate non-object memory (like
the hash table of a dict or the array of pointers for a list).
According to a comment in pymem.h, for allocating non-object memory,
you should always use the PyMem_* family of functions and macros.  In
addition, it is also quite clear on the need for extension modules to
always use the functions, not the macros.

> Much of a pymalloc fan as I am, I'm -1 on introducing it if we can't
> give extension authors a clear, brief, prominent and self-contained
> account of the memory rules they're supposed to follow (I don't
> really know what they are -- I always reverse-engineer it as needed,
> because I never found a place in the docs that offered sufficient
> guidance).

I think the problem is not that there are no clear unambiguous rules.
It's that they have been undocumented and violated in code used as an
example for too long.

We should at least try to do something in 2.3 that will help extension
authors do the right thing in subsequent versions.  Documenting the
rules is one thing we should definitely do.  Maybe we can also provide
a configuration option that complains loudly when an extension is
compiled that uses the wrong macros.  Maybe that configuration option
could even be on by default (it should only affect *compiling*
extensions, not linking with them).

--Guido van Rossum (home page: http://www.python.org/~guido/)