Suggested PyMem & PyObject_NEW includes

[Vladimir Marangozov]

Sorry for the delay -- I simply could't progress on this as I wanted to.
Here's the includes I suggest for PyMem and PyObject_New cs.

PyMem is okay. Some questions arise with PyObject_NEW.

1) I'm willing to unify the implementation on Windows and Unix
   (so I'm retaining the Windows variant of _PyObject_New
    reincarnated by PyObject_FromType -- see the comments in pyobjimpl.h).

2) For the user, there's the principle to use functions if
   binary compatibility is desired, and macros if she needs
   to trade compatibility for speed.

   But there's the issue of allocating the user objects with
   PyMem, or, allocate the objects with a custom allocator.

   After scratching my head on how to preserve bin compatibility with
   old libraries and offer the freedom to the user, I ended with the
   following (subject to discussion):

   - Use the functions for bin compat (but have also an exception
     _PyObject_Del(), with a leading underscore, for the core...)
     Objects in this case are allocated with PyMem.

   - Use the macros for allocating the objects with the potentially
     custom allocator (through malloc, realloc, free -- see below)

What do you think?

-----------------------------[ mymalloc.h ]---------------------------
...

/*
 * Core memory allocator
 * =====================
 */

/* To make sure the interpreter is user-malloc friendly, all memory
   and object APIs are implemented on top of this one.

   The PyCore_* macros can be changed to make the interpreter use a
   custom allocator. Note that they are for internal use only. Both
   the core and extension modules should use the PyMem_* API. */

#define PyCore_MALLOC_FUNC	malloc
#define PyCore_REALLOC_FUNC	realloc
#define PyCore_FREE_FUNC	free

#define PyCore_MALLOC_PROTO	Py_PROTO((size_t))
#define PyCore_REALLOC_PROTO	Py_PROTO((ANY *, size_t))
#define PyCore_FREE_PROTO	Py_PROTO((ANY *))

#define PyCore_MALLOC(n)       	PyCore_MALLOC_FUNC(n)
#define PyCore_REALLOC(p, n)	PyCore_REALLOC_FUNC((p), (n))
#define PyCore_FREE(p)		PyCore_FREE_FUNC(p)

/* The following should never be necessary */
#ifdef NEED_TO_DECLARE_MALLOC_AND_FRIEND
extern ANY *PyCore_MALLOC_FUNC PyCore_MALLOC_PROTO;
extern ANY *PyCore_REALLOC_FUNC PyCore_REALLOC_PROTO;
extern void PyCore_FREE_FUNC PyCore_FREE_PROTO;
#endif

/* BEWARE:

   Each interface exports both functions and macros. Extension modules
   should normally use the functions for ensuring binary compatibility
   of the user's code across Python versions. Subsequently, if Python
   switches to its own malloc (different from standard malloc), no
   recompilation is required for the extensions.

   The macro versions trade compatibility for speed. They can be used
   whenever there is a performance problem, but their use implies
   recompilation of the code for each new Python release. The Python
   core uses the macros because it *is* compiled on every upgrade.
   This might not be the case with 3rd party extensions in a custom
   setup (for example, a customer does not always have access to the
   source of 3rd party deliverables). You have been warned! */

/*
 * Raw memory interface
 * ====================
 */

/* Functions */

/* Two sets of function wrappers around malloc and friends; useful if
   you need to be sure that you are using the same memory allocator as
   Python.  Note that the wrappers make sure that allocating 0 bytes
   returns a non-NULL pointer, even if the underlying malloc doesn't. */

/* These wrappers around malloc call PyErr_NoMemory() on failure */
extern DL_IMPORT(ANY *) Py_Malloc Py_PROTO((size_t));
extern DL_IMPORT(ANY *) Py_Realloc Py_PROTO((ANY *, size_t));
extern DL_IMPORT(void) Py_Free Py_PROTO((ANY *));

/* These wrappers around malloc *don't* call anything on failure */
extern DL_IMPORT(ANY *) PyMem_Malloc Py_PROTO((size_t));
extern DL_IMPORT(ANY *) PyMem_Realloc Py_PROTO((ANY *, size_t));
extern DL_IMPORT(void) PyMem_Free Py_PROTO((ANY *));

/* Macros */
#define PyMem_MALLOC(n)		PyCore_MALLOC(n)
#define PyMem_REALLOC(p, n)	PyCore_REALLOC((ANY *)(p), (n))
#define PyMem_FREE(p)		PyCore_FREE((ANY *)(p))

/*
 * Type-oriented memory interface
 * ==============================
 */

/* Functions */
#define PyMem_New(type, n) \
	( (type *) PyMem_Malloc((n) * sizeof(type)) )
#define PyMem_Resize(p, type, n) \
	( (p) = (type *) PyMem_Realloc((n) * sizeof(type)) )
#define PyMem_Del(p) PyMem_Free(p)

/* Macros */
#define PyMem_NEW(type, n) \
	( (type *) PyMem_MALLOC(_PyMem_EXTRA + (n) * sizeof(type)) )
#define PyMem_RESIZE(p, type, n) \
	if ((p) == NULL) \
		(p) = (type *) PyMem_MALLOC( \
				    _PyMem_EXTRA + (n) * sizeof(type)); \
	else \
		(p) = (type *) PyMem_REALLOC((p), \
				    _PyMem_EXTRA + (n) * sizeof(type))
#define PyMem_DEL(p) PyMem_FREE(p)

/* PyMem_XDEL is deprecated. To avoid the call when p is NULL,
   it's recommended to write the test explicitely in the code.
   Note that according to ANSI C, free(NULL) has no effect. */
#define PyMem_XDEL(p) if ((p) == NULL) ; else PyMem_DEL(p)

...


-----------------------------[ mymalloc.h ]---------------------------
...

/*
Functions and macros for modules that implement new object types.
You must first include "object.h".

PyObject_New(type, typeobj) allocates memory for a new object of the given
type; here 'type' must be the C structure type used to represent the
object and 'typeobj' the address of the corresponding type object.
Reference count and type pointer are filled in; the rest of the bytes of
the object are *undefined*!  The resulting expression type is 'type *'.
The size of the object is actually determined by the tp_basicsize field
of the type object.

PyObject_NewVar(type, typeobj, n) is similar but allocates a variable-size
object with n extra items.  The size is computed as tp_basicsize plus
n * tp_itemsize.  This fills in the ob_size field as well.

PyObject_Del(op) releases the memory allocated for an object.

Two versions of the object constructors/destructors are provided:

1) PyObject_{New, NewVar, Del} delegate the allocation of the objects
   to the Python allocator which places them within the bounds of the
   Python heap. This way, Python keeps control on the user's objects
   regarding their memory management; for instance, they may be subject
   to automatic garbage collection, once their reference count drops to
   zero. Binary compatibility is preserved and there's no need to
   recompile the extension every time a new Python release comes out.

2) PyObject_{NEW, NEW_VAR, DEL} use the allocator of the extension module
   which *may* differ from the one used by the Python library. Typically,
   in a C++ module one may wish to redefine the default allocation
   strategy by overloading the operators new and del. In this case,
   however, the extension does not cooperate with the Python memory
   manager. The latter has no control on the user's objects as they won't
   be allocated within the Python heap. Therefore, automatic garbage
   collection may not be performed, binary compatibility is not
   guaranteed and recompilation is required on every new Python release.

Unless a specific memory management is needed, it's recommended to use 1).
*/


/*
In pre-Python-1.6 times, only the PyObject_{NEW, NEW_VAR} macros were
defined in terms of internal functions _PyObject_{New, NewVar}, the
implementation of which used to differ for Windows and non-Windows
platforms (see object.c -- these functions are left for backwards
compatibility with old libraries).

Starting from 1.6, an unified interface was introduced for both 1) & 2)
*/

extern DL_IMPORT(PyObject *) PyObject_FromType Py_PROTO((PyTypeObject *, PyObject *));
extern DL_IMPORT(PyVarObject *) PyObject_VarFromType Py_PROTO((PyTypeObject *, int, PyVarObject *));
extern DL_IMPORT(void) PyObject_Del Py_PROTO((PyObject *));

/* Functions */
#define PyObject_New(type, typeobj) \
		((type *) PyObject_FromType(typeobj, NULL))
#define PyObject_NewVar(type, typeobj, n) \
		((type *) PyObject_VarFromType((typeobj), (n), NULL))
#define PyObject_Del(op) PyObject_Del((PyObject *)(op))

/* XXX This trades binary compatibility for speed. */
#include "mymalloc.h"
#define _PyObject_Del(op) PyMem_FREE((PyObject *)(op))

/* Macros */
#define PyObject_NEW(type, typeobj) \
		((type *) PyObject_FromType(typeobj, \
			(PyObject *) malloc((typeobj)->tp_basicsize)))
#define PyObject_NEW_VAR(type, typeobj, n) \
		((type *) PyObject_VarFromType(typeobj, \
			(PyVarObject *) malloc((typeobj)->tp_basicsize + \
						n * (typeobj)->tp_itemsize)))
#define PyObject_DEL(op) free(op)

----------------------------------------------------------------------

So with this, I'm planning to "give the example" by renaming everywhere
in the distrib PyObject_NEW with PyObject_New, but use for the core
_PyObject_Del instead of PyObject_Del. I'll use PyObject_Del for the
objects defined in extension modules.

The point is that I don't want to define PyObject_Del in terms of
PyMem_FREE (or define PyObject_New in terms of PyMem_MALLOC) as this
would break the principle of binary compatibility when the Python
allocator is changed to a custom malloc from one build to another.

OTOH, I don't like the underscore...

Do you have a better suggestion?

-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252


-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov@inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252