[Patches] PyMem [1/8] - intro

[Guido van Rossum]

OK, in my copious spare time (TM Larry Wall) I had a good look at
Vladimir's malloc patches.  I'm torn.  I like a lot of the changes,
but I don't like having two new files.  Especially the new include
file "for internal use only" seriously bothers me: people are going to
ignore the warning.  There are no precedents: all files in the Include
directory are for public use (and most are included by Python.h); all
private files live in one of the other source directories (Parser,
Objects, Python, Modules).

I also dislike the fact that extensions are made second-class citizens
who don't have a macro that expands directly to malloc().

I think I'd prefer to have the following set of APIs:

Py_Malloc c.s.: unchanged (calls PyMem_MALLOC c.s., may call PyErr_NoMemory)
_PyObject_New c.s.: unchanged
PyObject_NEW c.s.: unchanged (#define using _PyObject_New c.s.)

PyMem_MALLOC c.s.: #defined as malloc c.s., normally
PyMem_Malloc c.s.: functions calling PyMem_MALLOC c.s.

PyMem_NEW c.s.: #defined using PyMem_MALLOC c.s.

I *like* the new PyObject_DEL and _PyObject_Del.  (I would have liked
to have them for my GC experiment, but that's another story.)

I can explain the MS_COREDLL business:

This is defined on Windows because the core is in a DLL.  Since the
caller may be in another DLL, and each DLL (potentially) has a
different default allocator, and (in pre-Vladimir times) the
type-specific deallocator typically calls free(), we (Mark & I)
decided that the allocation should be done in the type-specific
allocator.  We changed the PyObject_NEW() macro to call malloc() and
pass that into _PyObject_New() as a second argument.

Looking back, I wish we had made this change unconditionally --
there's nothing wrong with doing it that way on Unix, too.

But now that Vladimir is fixing all type-specific deallocators to call
the new _PyObject_Del(), we can put the allocation back into
_PyObject_New()!

The only problem is compatibility with old DLLs on Windows.  One
solution is to keep the second argument to _PyObject_New() when
MS_COREDLL is defined, but to change the PyObject_NEW() macro to pass
NULL instead.  The code of _PyObject_New() checks whether the pointer
is NULL and then allocates the memory itself; if a non-NULL pointer is
passed, it uses that and skips its own allocation.  _PyObject_Del()
can unconditionally free the memory: old DLLs won't call it (they call
free() directly) and new DLLs will have used NULL for the second
_PyObject_New() argument.

I hate to ask this of Vladimir, but could you make a new patch set
along the lines I sketched, without a pycore.h file?

--Guido van Rossum (home page: http://www.python.org/~guido/)

> From: Vladimir Marangozov <marangoz@python.inrialpes.fr>
> 
> I'm about to send a couple of patches relative to the malloc cleanup
> discussed on python-dev. Since it affects many files in the distribution,
> I'll send a patch per directory, which will facilitate their review.
> 
> Please examine them carefully. Although I'm pretty confident in their
> correctness, some details may have escaped my eyes due to the volume
> of the modified code.
> 
> Summary:
> ========
> 
> Exported (public) interfaces
> ----------------------------
> 
>    - PyMem_{Malloc, Realloc, Free} -- don't call anything on failure
>    - Py_{Malloc, Realloc, Free}    -- call PyErr_NoMemory() on failure
>    - _PyObject_{New, NewVar, Del}  -- object constructors/destructor
> 
>    Macros:
>    - PyMem_{NEW, NEW_VAR, DEL}     -- handy, use PyMem_{Malloc...} above
>    - PyObject_{NEW, NEW_VAR, DEL}  -- handy, use _PyObject_{New...} above
> 
> Private core interfaces
> -----------------------
> 
>    In addition to the public ones:
> 
>    - PyMem_{MALLOC, REALLOC, FREE} -- Python core allocator
>    
>    Macros:
>    - PyMem_{NEW, NEW_VAR, DEL}     -- handy, use PyMem_{MALLOC...} above
>    - PyObject_DEL is inlined (PyMem_FREE)