[Python-Dev] RE: Program very slow to finish

[Tim Peters]

[Neil Schemenauer]
> The problem is with extension modules.  We can make sure code in CVS
> always has the big lock held when calling PyMalloc.  We can't be sure
> that extension modules are safe.

So can that *ever* be solved?  Some part of the puzzle went unaddressed
here.

> The other, more serious problem is that many extension modules allocate
> memory with PyObject_New() and free it with free() or PyMem_DEL()
> instead of PyObject_Del() or PyObject_DEL().

Umm, OK, PyObject_DEL() ... is a macro ... which redirects to
PyObject_FREE() ... which is a macro ... which redirects to
PyCore_OBJECT_FREE() ... which is a macro ... which redirects to
PyCore_OBJECT_FREE_FUNC() ... which is a macro ... which redirects to
_PyCore_ObjectFree (WITH_PYMALLOC) or PyCore_FREE_FUNC (without).
PyCore_FREE_FUNC is a macro ... which redirects to free().  And
_PyCore_ObjectFree ... doesn't exist.  I must have missed an #undef in there
somewhere ... ah, OK, in the WITH_PYMALLOC case, PyCore_OBJECT_FREE_FUNC(==
_PyCore_ObjectFree) appears in obmalloc.c's

#define _THIS_FREE		PyCore_OBJECT_FREE_FUNC

so that _THIS_FREE is actually _PyCore_ObjectFree; then obmalloc.c's

void
_THIS_FREE(void *p)
{

expands to

void
PyCore_OBJECT_FREE_FUNC(void *p)
{

which expands again to

void
_PyCore_ObjectFree(void *p)
{

and we're done.  Couldn't be simpler <wink>.  Then PyMem_DEL ... oh, forget
it.  Now I remember why I gave up on this last time I looked at it -- it's
Preprocessor Hell.

> If pymalloc is enabled then memory allocated by a PyObject_* function
> must be freed by a PyObject_* funciton.

Damn, you're good.

> mxDateTime is the first module I ran into that does this but there are
> many others I'm sure.  I think almost all of the C modules I have
> written do it.  The code in xxmodule.c used to do it as well.
>
>  no-solutions-only-more-problems-hey-i'm-sick-today-too-ly y'rs Neil

Big Hammer?  Change every one of the existing guys to resolve to malloc()
and free().  Then declare them all obsolete, define a much smaller new set
of names, edit the core to use the new guys, and non-core modules get stuck
with malloc/free until they're rewritten too.  It sucks, but we'd be in
better shape today if that *had* been done the last time this got reworked.