[Python-checkins] cpython: Revert changeset 6661a8154eb3: Issue #3329: Add new APIs to customize memory
victor.stinner
python-checkins at python.org
Sat Jun 15 03:38:03 CEST 2013
http://hg.python.org/cpython/rev/b1455dd08000
changeset: 84131:b1455dd08000
parent: 84128:5619bc2d8207
user: Victor Stinner <victor.stinner at gmail.com>
date: Sat Jun 15 03:37:01 2013 +0200
summary:
Revert changeset 6661a8154eb3: Issue #3329: Add new APIs to customize memory allocators
The new API require more discussion.
files:
Doc/c-api/memory.rst | 121 +------
Include/objimpl.h | 71 +-
Include/pymem.h | 91 +---
Modules/_testcapimodule.c | 178 ---------
Objects/object.c | 20 +
Objects/obmalloc.c | 501 +++++++------------------
6 files changed, 211 insertions(+), 771 deletions(-)
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -84,46 +84,6 @@
for the I/O buffer escapes completely the Python memory manager.
-Raw Memory Interface
-====================
-
-The following function are wrappers to system allocators: :c:func:`malloc`,
-:c:func:`realloc`, :c:func:`free`. These functions are thread-safe, the
-:term:`GIL <global interpreter lock>` does not need to be held to use these
-functions.
-
-The behaviour of requesting zero bytes is not defined: return *NULL* or a
-distinct non-*NULL* pointer depending on the platform. Use
-:c:func:`PyMem_Malloc` and :c:func:`PyMem_Realloc` to have a well defined
-behaviour.
-
-.. versionadded:: 3.4
-
-.. c:function:: void* PyMem_RawMalloc(size_t n)
-
- Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
- allocated memory, or *NULL* if the request fails. The memory
- will not have been initialized in any way.
-
-
-.. c:function:: void* PyMem_RawRealloc(void *p, size_t n)
-
- Resizes the memory block pointed to by *p* to *n* bytes. The contents will
- be unchanged to the minimum of the old and the new sizes. If *p* is *NULL*,
- the call is equivalent to ``PyMem_RawMalloc(n)``. Unless *p* is *NULL*, it
- must have been returned by a previous call to :c:func:`PyMem_RawMalloc` or
- :c:func:`PyMem_RawRealloc`. If the request fails, :c:func:`PyMem_RawRealloc`
- returns *NULL* and *p* remains a valid pointer to the previous memory area.
-
-
-.. c:function:: void PyMem_RawFree(void *p)
-
- Frees the memory block pointed to by *p*, which must have been returned by a
- previous call to :c:func:`PyMem_RawMalloc` or :c:func:`PyMem_RawRealloc`.
- Otherwise, or if ``PyMem_Free(p)`` has been called before, undefined
- behavior occurs. If *p* is *NULL*, no operation is performed.
-
-
.. _memoryinterface:
Memory Interface
@@ -131,12 +91,8 @@
The following function sets, modeled after the ANSI C standard, but specifying
behavior when requesting zero bytes, are available for allocating and releasing
-memory from the Python heap.
+memory from the Python heap:
-.. warning::
-
- The :term:`GIL <global interpreter lock>` must be held when using these
- functions.
.. c:function:: void* PyMem_Malloc(size_t n)
@@ -199,81 +155,6 @@
:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`.
-Customize Memory Allocators
-===========================
-
-.. versionadded:: 3.4
-
-.. c:type:: PyMemAllocators
-
- Structure used to describe memory allocator. This structure has
- four fields:
-
- +----------------------------------------------------------+-----------------+
- | Field | Meaning |
- +==========================================================+=================+
- | ``void *ctx`` | user data |
- +----------------------------------------------------------+-----------------+
- | ``void* malloc(void *ctx, size_t size)`` | allocate memory |
- +----------------------------------------------------------+-----------------+
- | ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate memory |
- | | or resize a |
- | | memory block |
- +----------------------------------------------------------+-----------------+
- | ``void free(void *ctx, void *ptr)`` | release memory |
- +----------------------------------------------------------+-----------------+
-
-.. c:function:: void PyMem_GetRawAllocators(PyMemAllocators *allocators)
-
- Get internal functions of :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`
- and :c:func:`PyMem_RawFree`.
-
-.. c:function:: void PyMem_SetRawAllocators(PyMemAllocators *allocators)
-
- Set internal functions of :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`
- and :c:func:`PyMem_RawFree`.
-
- :c:func:`PyMem_SetupDebugHooks` should be called to reinstall debug hooks if
- new functions do no call original functions anymore.
-
-.. c:function:: void PyMem_GetAllocators(PyMemAllocators *allocators)
-
- Get internal functions of :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`
- and :c:func:`PyMem_Free`.
-
-.. c:function:: void PyMem_SetAllocators(PyMemAllocators *allocators)
-
- Set internal functions of :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`
- and :c:func:`PyMem_Free`.
-
- ``malloc(ctx, 0)`` and ``realloc(ctx, ptr, 0)`` must not return *NULL*: it
- would be treated as an error.
-
- :c:func:`PyMem_SetupDebugHooks` should be called to reinstall debug hooks if
- new functions do no call original functions anymore.
-
-.. c:function:: void PyMem_SetupDebugHooks(void)
-
- Setup hooks to detect bugs in the following Python memory allocator
- functions:
-
- - :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc`,
- :c:func:`PyMem_RawFree`
- - :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc`, :c:func:`PyMem_Free`
- - :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc`,
- :c:func:`PyObject_Free`
-
- Newly allocated memory is filled with the byte ``0xCB``, freed memory is
- filled with the byte ``0xDB``. Additionnal checks:
-
- - detect API violations, ex: :c:func:`PyObject_Free` called on a buffer
- allocated by :c:func:`PyMem_Malloc`
- - detect write before the start of the buffer (buffer underflow)
- - detect write after the end of the buffer (buffer overflow)
-
- The function does nothing if Python is not compiled is debug mode.
-
-
.. _memoryexamples:
Examples
diff --git a/Include/objimpl.h b/Include/objimpl.h
--- a/Include/objimpl.h
+++ b/Include/objimpl.h
@@ -94,9 +94,9 @@
the object gets initialized via PyObject_{Init, InitVar} after obtaining
the raw memory.
*/
-PyAPI_FUNC(void *) PyObject_Malloc(size_t size);
-PyAPI_FUNC(void *) PyObject_Realloc(void *ptr, size_t new_size);
-PyAPI_FUNC(void) PyObject_Free(void *ptr);
+PyAPI_FUNC(void *) PyObject_Malloc(size_t);
+PyAPI_FUNC(void *) PyObject_Realloc(void *, size_t);
+PyAPI_FUNC(void) PyObject_Free(void *);
/* This function returns the number of allocated memory blocks, regardless of size */
PyAPI_FUNC(Py_ssize_t) _Py_GetAllocatedBlocks(void);
@@ -106,46 +106,41 @@
#ifndef Py_LIMITED_API
PyAPI_FUNC(void) _PyObject_DebugMallocStats(FILE *out);
#endif /* #ifndef Py_LIMITED_API */
-#endif
+#ifdef PYMALLOC_DEBUG /* WITH_PYMALLOC && PYMALLOC_DEBUG */
+PyAPI_FUNC(void *) _PyObject_DebugMalloc(size_t nbytes);
+PyAPI_FUNC(void *) _PyObject_DebugRealloc(void *p, size_t nbytes);
+PyAPI_FUNC(void) _PyObject_DebugFree(void *p);
+PyAPI_FUNC(void) _PyObject_DebugDumpAddress(const void *p);
+PyAPI_FUNC(void) _PyObject_DebugCheckAddress(const void *p);
+PyAPI_FUNC(void *) _PyObject_DebugMallocApi(char api, size_t nbytes);
+PyAPI_FUNC(void *) _PyObject_DebugReallocApi(char api, void *p, size_t nbytes);
+PyAPI_FUNC(void) _PyObject_DebugFreeApi(char api, void *p);
+PyAPI_FUNC(void) _PyObject_DebugCheckAddressApi(char api, const void *p);
+PyAPI_FUNC(void *) _PyMem_DebugMalloc(size_t nbytes);
+PyAPI_FUNC(void *) _PyMem_DebugRealloc(void *p, size_t nbytes);
+PyAPI_FUNC(void) _PyMem_DebugFree(void *p);
+#define PyObject_MALLOC _PyObject_DebugMalloc
+#define PyObject_Malloc _PyObject_DebugMalloc
+#define PyObject_REALLOC _PyObject_DebugRealloc
+#define PyObject_Realloc _PyObject_DebugRealloc
+#define PyObject_FREE _PyObject_DebugFree
+#define PyObject_Free _PyObject_DebugFree
-/* Macros */
+#else /* WITH_PYMALLOC && ! PYMALLOC_DEBUG */
#define PyObject_MALLOC PyObject_Malloc
#define PyObject_REALLOC PyObject_Realloc
#define PyObject_FREE PyObject_Free
+#endif
+
+#else /* ! WITH_PYMALLOC */
+#define PyObject_MALLOC PyMem_MALLOC
+#define PyObject_REALLOC PyMem_REALLOC
+#define PyObject_FREE PyMem_FREE
+
+#endif /* WITH_PYMALLOC */
+
#define PyObject_Del PyObject_Free
-#define PyObject_DEL PyObject_Free
-
-/* Get internal functions of PyObject_Malloc(), PyObject_Realloc() and
- PyObject_Free(). *ctx_p is an arbitrary user value. */
-PyAPI_FUNC(void) PyObject_GetAllocators(PyMemAllocators *allocators);
-
-/* Set internal functions of PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
- ctx is an arbitrary user value.
-
- malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return NULL: it would be
- treated as an error.
-
- PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
- functions do no call original functions anymore. */
-PyAPI_FUNC(void) PyObject_SetAllocators(PyMemAllocators *allocators);
-
-/* Get internal functions allocating and deallocating arenas for
- PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
- *ctx_p is an arbitrary user value. */
-PyAPI_FUNC(void) _PyObject_GetArenaAllocators(
- void **ctx_p,
- void* (**malloc_p) (void *ctx, size_t size),
- void (**free_p) (void *ctx, void *ptr, size_t size)
- );
-
-/* Get internal functions allocating and deallocating arenas for
- PyObject_Malloc(), PyObject_Realloc() and PyObject_Free().
- ctx is an arbitrary user value. */
-PyAPI_FUNC(void) _PyObject_SetArenaAllocators(
- void *ctx,
- void* (*malloc) (void *ctx, size_t size),
- void (*free) (void *ctx, void *ptr, size_t size)
- );
+#define PyObject_DEL PyObject_FREE
/*
* Generic object allocator interface
diff --git a/Include/pymem.h b/Include/pymem.h
--- a/Include/pymem.h
+++ b/Include/pymem.h
@@ -11,40 +11,6 @@
extern "C" {
#endif
-typedef struct {
- /* user context passed as the first argument to the 3 functions */
- void *ctx;
-
- /* allocate memory */
- void* (*malloc) (void *ctx, size_t size);
-
- /* allocate memory or resize a memory buffer */
- void* (*realloc) (void *ctx, void *ptr, size_t new_size);
-
- /* release memory */
- void (*free) (void *ctx, void *ptr);
-} PyMemAllocators;
-
-/* Raw memory allocators, system functions: malloc(), realloc(), free().
-
- These functions are thread-safe, the GIL does not need to be held. */
-
-/* Get internal functions of PyMem_RawMalloc(), PyMem_RawRealloc() and
- PyMem_RawFree(). *ctx_p is an arbitrary user value. */
-PyAPI_FUNC(void) PyMem_GetRawAllocators(PyMemAllocators *allocators);
-
-/* Set internal functions of PyMem_RawMalloc(), PyMem_RawRealloc() and
- PyMem_RawFree(). ctx is an arbitrary user value.
-
- PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
- functions do no call original functions anymore. */
-PyAPI_FUNC(void) PyMem_SetRawAllocators(PyMemAllocators *allocators);
-
-PyAPI_FUNC(void *) PyMem_RawMalloc(size_t size);
-PyAPI_FUNC(void *) PyMem_RawRealloc(void *ptr, size_t new_size);
-PyAPI_FUNC(void) PyMem_RawFree(void *ptr);
-
-
/* BEWARE:
Each interface exports both functions and macros. Extension modules should
@@ -83,11 +49,21 @@
performed on failure (no exception is set, no warning is printed, etc).
*/
-PyAPI_FUNC(void *) PyMem_Malloc(size_t size);
-PyAPI_FUNC(void *) PyMem_Realloc(void *ptr, size_t new_size);
-PyAPI_FUNC(void) PyMem_Free(void *ptr);
+PyAPI_FUNC(void *) PyMem_Malloc(size_t);
+PyAPI_FUNC(void *) PyMem_Realloc(void *, size_t);
+PyAPI_FUNC(void) PyMem_Free(void *);
+
+/* Starting from Python 1.6, the wrappers Py_{Malloc,Realloc,Free} are
+ no longer supported. They used to call PyErr_NoMemory() on failure. */
/* Macros. */
+#ifdef PYMALLOC_DEBUG
+/* Redirect all memory operations to Python's debugging allocator. */
+#define PyMem_MALLOC _PyMem_DebugMalloc
+#define PyMem_REALLOC _PyMem_DebugRealloc
+#define PyMem_FREE _PyMem_DebugFree
+
+#else /* ! PYMALLOC_DEBUG */
/* PyMem_MALLOC(0) means malloc(1). Some systems would return NULL
for malloc(0), which would be treated as an error. Some platforms
@@ -95,9 +71,13 @@
pymalloc. To solve these problems, allocate an extra byte. */
/* Returns NULL to indicate error if a negative size or size larger than
Py_ssize_t can represent is supplied. Helps prevents security holes. */
-#define PyMem_MALLOC(n) PyMem_Malloc(n)
-#define PyMem_REALLOC(p, n) PyMem_Realloc(p, n)
-#define PyMem_FREE(p) PyMem_Free(p)
+#define PyMem_MALLOC(n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \
+ : malloc((n) ? (n) : 1))
+#define PyMem_REALLOC(p, n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \
+ : realloc((p), (n) ? (n) : 1))
+#define PyMem_FREE free
+
+#endif /* PYMALLOC_DEBUG */
/*
* Type-oriented memory interface
@@ -135,37 +115,6 @@
#define PyMem_Del PyMem_Free
#define PyMem_DEL PyMem_FREE
-/* Get internal functions of PyMem_Malloc(), PyMem_Realloc()
- and PyMem_Free() */
-PyAPI_FUNC(void) PyMem_GetAllocators(PyMemAllocators *allocators);
-
-/* Set internal functions of PyMem_Malloc(), PyMem_Realloc() and PyMem_Free().
-
- malloc(ctx, 0) and realloc(ctx, ptr, 0) must not return NULL: it would be
- treated as an error.
-
- PyMem_SetupDebugHooks() should be called to reinstall debug hooks if new
- functions do no call original functions anymore. */
-PyAPI_FUNC(void) PyMem_SetAllocators(PyMemAllocators *allocators);
-
-/* Setup hooks to detect bugs in the following Python memory allocator
- functions:
-
- - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree()
- - PyMem_Malloc(), PyMem_Realloc(), PyMem_Free()
- - PyObject_Malloc(), PyObject_Realloc() and PyObject_Free()
-
- Newly allocated memory is filled with the byte 0xCB, freed memory is filled
- with the byte 0xDB. Additionnal checks:
-
- - detect API violations, ex: PyObject_Free() called on a buffer allocated
- by PyMem_Malloc()
- - detect write before the start of the buffer (buffer underflow)
- - detect write after the end of the buffer (buffer overflow)
-
- The function does nothing if Python is not compiled is debug mode. */
-PyAPI_FUNC(void) PyMem_SetupDebugHooks(void);
-
#ifdef __cplusplus
}
#endif
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2511,176 +2511,6 @@
Py_RETURN_NONE;
}
-static PyObject *
-test_pymem_alloc0(PyObject *self)
-{
- void *ptr;
-
- ptr = PyMem_Malloc(0);
- if (ptr == NULL) {
- PyErr_SetString(PyExc_RuntimeError, "PyMem_Malloc(0) returns NULL");
- return NULL;
- }
- PyMem_Free(ptr);
-
- ptr = PyObject_Malloc(0);
- if (ptr == NULL) {
- PyErr_SetString(PyExc_RuntimeError, "PyObject_Malloc(0) returns NULL");
- return NULL;
- }
- PyObject_Free(ptr);
-
- Py_RETURN_NONE;
-}
-
-typedef struct {
- PyMemAllocators alloc;
-
- size_t malloc_size;
- void *realloc_ptr;
- size_t realloc_new_size;
- void *free_ptr;
-} alloc_hook_t;
-
-static void* hook_malloc (void* ctx, size_t size)
-{
- alloc_hook_t *hook = (alloc_hook_t *)ctx;
- hook->malloc_size = size;
- return hook->alloc.malloc(hook->alloc.ctx, size);
-}
-
-static void* hook_realloc (void* ctx, void* ptr, size_t new_size)
-{
- alloc_hook_t *hook = (alloc_hook_t *)ctx;
- hook->realloc_ptr = ptr;
- hook->realloc_new_size = new_size;
- return hook->alloc.realloc(hook->alloc.ctx, ptr, new_size);
-}
-
-static void hook_free (void *ctx, void *ptr)
-{
- alloc_hook_t *hook = (alloc_hook_t *)ctx;
- printf("HOOK\n");
- hook->free_ptr = ptr;
- hook->alloc.free(hook->alloc.ctx, ptr);
-}
-
-static PyObject *
-test_setallocators(char api)
-{
- PyObject *res = NULL;
- const char *error_msg;
- alloc_hook_t hook;
- PyMemAllocators alloc;
- size_t size, size2;
- void *ptr, *ptr2;
-
- hook.malloc_size = 0;
- hook.realloc_ptr = NULL;
- hook.realloc_new_size = 0;
- hook.free_ptr = NULL;
-
- alloc.ctx = &hook;
- alloc.malloc = &hook_malloc;
- alloc.realloc = &hook_realloc;
- alloc.free = &hook_free;
- if (api == 'o') {
- PyObject_GetAllocators(&hook.alloc);
- PyObject_SetAllocators(&alloc);
- }
- else if (api == 'r') {
- PyMem_GetRawAllocators(&hook.alloc);
- PyMem_SetRawAllocators(&alloc);
- }
- else {
- PyMem_GetAllocators(&hook.alloc);
- PyMem_SetAllocators(&alloc);
- }
-
- size = 42;
- if (api == 'o')
- ptr = PyObject_Malloc(size);
- else if (api == 'r')
- ptr = PyMem_RawMalloc(size);
- else
- ptr = PyMem_Malloc(size);
- if (ptr == NULL) {
- error_msg = "malloc failed";
- goto fail;
- }
-
- if (hook.malloc_size != size) {
- error_msg = "malloc invalid size";
- goto fail;
- }
-
- size2 = 200;
- if (api == 'o')
- ptr2 = PyObject_Realloc(ptr, size2);
- else if (api == 'r')
- ptr2 = PyMem_RawRealloc(ptr, size2);
- else
- ptr2 = PyMem_Realloc(ptr, size2);
- if (ptr2 == NULL) {
- error_msg = "realloc failed";
- goto fail;
- }
-
- if (hook.realloc_ptr != ptr
- || hook.realloc_new_size != size2) {
- error_msg = "realloc invalid parameters";
- goto fail;
- }
-
- if (api == 'o')
- PyObject_Free(ptr2);
- else if (api == 'r')
- PyMem_RawFree(ptr2);
- else {
- printf("PyMem_Free\n");
- PyMem_Free(ptr2);
- }
-
- if (hook.free_ptr != ptr2) {
- error_msg = "free invalid pointer";
- goto fail;
- }
-
- Py_INCREF(Py_None);
- res = Py_None;
- goto finally;
-
-fail:
- PyErr_SetString(PyExc_RuntimeError, error_msg);
-
-finally:
- if (api == 'o')
- PyObject_SetAllocators(&hook.alloc);
- else if (api == 'r')
- PyMem_SetRawAllocators(&hook.alloc);
- else
- PyMem_SetAllocators(&hook.alloc);
- return res;
-}
-
-static PyObject *
-test_pymem_setrawallocators(PyObject *self)
-{
- return test_setallocators('r');
-}
-
-static PyObject *
-test_pymem_setallocators(PyObject *self)
-{
- return test_setallocators('m');
-}
-
-static PyObject *
-test_pyobject_setallocators(PyObject *self)
-{
- return test_setallocators('o');
-}
-
static PyMethodDef TestMethods[] = {
{"raise_exception", raise_exception, METH_VARARGS},
{"raise_memoryerror", (PyCFunction)raise_memoryerror, METH_NOARGS},
@@ -2781,14 +2611,6 @@
{"pytime_object_to_time_t", test_pytime_object_to_time_t, METH_VARARGS},
{"pytime_object_to_timeval", test_pytime_object_to_timeval, METH_VARARGS},
{"pytime_object_to_timespec", test_pytime_object_to_timespec, METH_VARARGS},
- {"test_pymem",
- (PyCFunction)test_pymem_alloc0, METH_NOARGS},
- {"test_pymem_alloc0",
- (PyCFunction)test_pymem_setrawallocators, METH_NOARGS},
- {"test_pymem_setallocators",
- (PyCFunction)test_pymem_setallocators, METH_NOARGS},
- {"test_pyobject_setallocators",
- (PyCFunction)test_pyobject_setallocators, METH_NOARGS},
{NULL, NULL} /* sentinel */
};
diff --git a/Objects/object.c b/Objects/object.c
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -1859,6 +1859,26 @@
Py_ssize_t (*_Py_abstract_hack)(PyObject *) = PyObject_Size;
+/* Python's malloc wrappers (see pymem.h) */
+
+void *
+PyMem_Malloc(size_t nbytes)
+{
+ return PyMem_MALLOC(nbytes);
+}
+
+void *
+PyMem_Realloc(void *p, size_t nbytes)
+{
+ return PyMem_REALLOC(p, nbytes);
+}
+
+void
+PyMem_Free(void *p)
+{
+ PyMem_FREE(p);
+}
+
void
_PyObject_DebugTypeStats(FILE *out)
{
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -1,327 +1,5 @@
#include "Python.h"
-/* Python's malloc wrappers (see pymem.h) */
-
-/* Forward declaration */
-
-#ifdef PYMALLOC_DEBUG /* WITH_PYMALLOC && PYMALLOC_DEBUG */
-static void* _PyMem_DebugMalloc(void *ctx, size_t size);
-static void _PyMem_DebugFree(void *ctx, void *p);
-static void* _PyMem_DebugRealloc(void *ctx, void *ptr, size_t size);
-
-static void _PyObject_DebugDumpAddress(const void *p);
-static void _PyMem_DebugCheckAddress(char api_id, const void *p);
-#endif
-
-#ifdef WITH_PYMALLOC
-static void* _PyObject_Malloc(void *ctx, size_t size);
-static void _PyObject_Free(void *ctx, void *p);
-static void* _PyObject_Realloc(void *ctx, void *ptr, size_t size);
-#endif
-
-
-static void *
-_PyMem_RawMalloc(void *ctx, size_t size)
-{
- return malloc(size);
-}
-
-static void *
-_PyMem_RawRealloc(void *ctx, void *ptr, size_t size)
-{
- return realloc(ptr, size);
-}
-
-static void
-_PyMem_RawFree(void *ctx, void *ptr)
-{
- return free(ptr);
-}
-
-static void *
-_PyMem_Malloc(void *ctx, size_t size)
-{
- /* PyMem_Malloc(0) means malloc(1). Some systems would return NULL
- for malloc(0), which would be treated as an error. Some platforms would
- return a pointer with no memory behind it, which would break pymalloc.
- To solve these problems, allocate an extra byte. */
- if (size == 0)
- size = 1;
- return malloc(size);
-}
-
-static void *
-_PyMem_Realloc(void *ctx, void *ptr, size_t size)
-{
- if (size == 0)
- size = 1;
- return realloc(ptr, size);
-}
-
-#ifdef ARENAS_USE_MMAP
-static void *
-_PyObject_ArenaMmap(void *ctx, size_t size)
-{
- void *ptr;
- ptr = mmap(NULL, size, PROT_READ|PROT_WRITE,
- MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
- if (ptr == MAP_FAILED)
- return NULL;
- assert(ptr != NULL);
- return ptr;
-}
-
-static void
-_PyObject_ArenaMunmap(void *ctx, void *ptr, size_t size)
-{
- return munmap(ptr, size);
-}
-#else
-static void *
-_PyObject_ArenaMalloc(void *ctx, size_t size)
-{
- return malloc(size);
-}
-
-static void
-_PyObject_ArenaFree(void *ctx, void *ptr, size_t size)
-{
- free(ptr);
-}
-#endif
-
-#define PYRAW_FUNCS _PyMem_RawMalloc, _PyMem_RawRealloc, _PyMem_RawFree
-#define PYMEM_FUNCS _PyMem_Malloc, _PyMem_Realloc, _PyMem_RawFree
-#ifdef WITH_PYMALLOC
-#define PYOBJECT_FUNCS _PyObject_Malloc, _PyObject_Realloc, _PyObject_Free
-#else
-#define PYOBJECT_FUNCS PYMEM_FUNCS
-#endif
-
-#ifdef PYMALLOC_DEBUG
-typedef struct {
- /* We tag each block with an API ID in order to tag API violations */
- char api_id;
- PyMemAllocators alloc;
-} debug_alloc_api_t;
-static struct {
- debug_alloc_api_t raw;
- debug_alloc_api_t mem;
- debug_alloc_api_t obj;
-} _PyMem_Debug = {
- {'r', {NULL, PYRAW_FUNCS}},
- {'m', {NULL, PYMEM_FUNCS}},
- {'o', {NULL, PYOBJECT_FUNCS}}
- };
-
-#define PYDEBUG_FUNCS _PyMem_DebugMalloc, _PyMem_DebugRealloc, _PyMem_DebugFree
-#endif
-
-static PyMemAllocators _PyMem_Raw = {
-#ifdef PYMALLOC_DEBUG
- &_PyMem_Debug.raw, PYDEBUG_FUNCS
-#else
- NULL, PYMEM_FUNCS
-#endif
- };
-
-static PyMemAllocators _PyMem = {
-#ifdef PYMALLOC_DEBUG
- &_PyMem_Debug.mem, PYDEBUG_FUNCS
-#else
- NULL, PYMEM_FUNCS
-#endif
- };
-
-static PyMemAllocators _PyObject = {
-#ifdef PYMALLOC_DEBUG
- &_PyMem_Debug.obj, PYDEBUG_FUNCS
-#else
- NULL, PYOBJECT_FUNCS
-#endif
- };
-
-#undef PYRAW_FUNCS
-#undef PYMEM_FUNCS
-#undef PYOBJECT_FUNCS
-#undef PYDEBUG_FUNCS
-
-static struct {
- void *ctx;
- void* (*malloc) (void*, size_t);
- void (*free) (void*, void*, size_t);
-} _PyObject_Arena = {NULL,
-#ifdef ARENAS_USE_MMAP
- _PyObject_ArenaMmap, _PyObject_ArenaMunmap
-#else
- _PyObject_ArenaMalloc, _PyObject_ArenaFree
-#endif
- };
-
-void
-PyMem_SetupDebugHooks(void)
-{
-#ifdef PYMALLOC_DEBUG
- PyMemAllocators alloc;
-
- alloc.malloc = _PyMem_DebugMalloc;
- alloc.realloc = _PyMem_DebugRealloc;
- alloc.free = _PyMem_DebugFree;
-
- if (_PyMem_Raw.malloc != _PyMem_DebugMalloc) {
- alloc.ctx = &_PyMem_Debug.raw;
- PyMem_GetAllocators(&_PyMem_Debug.raw.alloc);
- PyMem_SetAllocators(&alloc);
- }
-
- if (_PyMem.malloc != _PyMem_DebugMalloc) {
- alloc.ctx = &_PyMem_Debug.mem;
- PyMem_GetAllocators(&_PyMem_Debug.mem.alloc);
- PyMem_SetAllocators(&alloc);
- }
-
- if (_PyObject.malloc != _PyMem_DebugMalloc) {
- alloc.ctx = &_PyMem_Debug.obj;
- PyObject_GetAllocators(&_PyMem_Debug.obj.alloc);
- PyObject_SetAllocators(&alloc);
- }
-#endif
-}
-
-void
-PyMem_GetRawAllocators(PyMemAllocators *allocators)
-{
- *allocators = _PyMem_Raw;
-}
-
-void
-PyMem_SetRawAllocators(PyMemAllocators *allocators)
-{
- _PyMem_Raw = *allocators;
-}
-
-void
-PyMem_GetAllocators(PyMemAllocators *allocators)
-{
- *allocators = _PyMem;
-}
-
-void
-PyMem_SetAllocators(PyMemAllocators *allocators)
-{
- _PyMem = *allocators;
-}
-
-void
-PyObject_GetAllocators(PyMemAllocators *allocators)
-{
- *allocators = _PyObject;
-}
-
-void
-PyObject_SetAllocators(PyMemAllocators *allocators)
-{
- _PyObject = *allocators;
-}
-
-void
-_PyObject_GetArenaAllocators(void **ctx_p,
- void* (**malloc_p) (void *ctx, size_t size),
- void (**free_p) (void *ctx, void *ptr, size_t size))
-{
- *malloc_p = _PyObject_Arena.malloc;
- *free_p = _PyObject_Arena.free;
- *ctx_p = _PyObject_Arena.ctx;
-}
-
-void
-_PyObject_SetArenaAllocators(void *ctx,
- void* (*malloc) (void *ctx, size_t size),
- void (*free) (void *ctx, void *ptr, size_t size))
-{
- _PyObject_Arena.malloc = malloc;
- _PyObject_Arena.free = free;
- _PyObject_Arena.ctx = ctx;
-}
-
-void *
-PyMem_RawMalloc(size_t size)
-{
- return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
-}
-
-void*
-PyMem_RawRealloc(void *ptr, size_t new_size)
-{
- return _PyMem_Raw.realloc(_PyMem_Raw.ctx, ptr, new_size);
-}
-
-void PyMem_RawFree(void *ptr)
-{
- _PyMem_Raw.free(_PyMem_Raw.ctx, ptr);
-}
-
-void *
-PyMem_Malloc(size_t size)
-{
- /*
- * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
- * Most python internals blindly use a signed Py_ssize_t to track
- * things without checking for overflows or negatives.
- * As size_t is unsigned, checking for size < 0 is not required.
- */
- if (size > (size_t)PY_SSIZE_T_MAX)
- return NULL;
-
- return _PyMem.malloc(_PyMem.ctx, size);
-}
-
-void *
-PyMem_Realloc(void *ptr, size_t new_size)
-{
- if (new_size > (size_t)PY_SSIZE_T_MAX)
- return NULL;
-
- return _PyMem.realloc(_PyMem.ctx, ptr, new_size);
-}
-
-void
-PyMem_Free(void *ptr)
-{
- _PyMem.free(_PyMem.ctx, ptr);
-}
-
-void *
-PyObject_Malloc(size_t size)
-{
- /*
- * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
- * Most python internals blindly use a signed Py_ssize_t to track
- * things without checking for overflows or negatives.
- * As size_t is unsigned, checking for size < 0 is not required.
- */
- if (size > (size_t)PY_SSIZE_T_MAX)
- return NULL;
-
- return _PyObject.malloc(_PyObject.ctx, size);
-}
-
-void *
-PyObject_Realloc(void *ptr, size_t new_size)
-{
- if (new_size > (size_t)PY_SSIZE_T_MAX)
- return NULL;
-
- return _PyObject.realloc(_PyObject.ctx, ptr, new_size);
-}
-
-void
-PyObject_Free(void *ptr)
-{
- _PyObject.free(_PyObject.ctx, ptr);
-}
-
-
#ifdef WITH_PYMALLOC
#ifdef HAVE_MMAP
@@ -867,6 +545,7 @@
struct arena_object* arenaobj;
uint excess; /* number of bytes above pool alignment */
void *address;
+ int err;
#ifdef PYMALLOC_DEBUG
if (Py_GETENV("PYTHONMALLOCSTATS"))
@@ -888,12 +567,11 @@
return NULL; /* overflow */
#endif
nbytes = numarenas * sizeof(*arenas);
- arenaobj = (struct arena_object *)PyMem_Realloc(arenas, nbytes);
+ arenaobj = (struct arena_object *)realloc(arenas, nbytes);
if (arenaobj == NULL)
return NULL;
arenas = arenaobj;
-
/* We might need to fix pointers that were copied. However,
* new_arena only gets called when all the pages in the
* previous arenas are full. Thus, there are *no* pointers
@@ -920,8 +598,15 @@
arenaobj = unused_arena_objects;
unused_arena_objects = arenaobj->nextarena;
assert(arenaobj->address == 0);
- address = _PyObject_Arena.malloc(_PyObject_Arena.ctx, ARENA_SIZE);
- if (address == NULL) {
+#ifdef ARENAS_USE_MMAP
+ address = mmap(NULL, ARENA_SIZE, PROT_READ|PROT_WRITE,
+ MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
+ err = (address == MAP_FAILED);
+#else
+ address = malloc(ARENA_SIZE);
+ err = (address == 0);
+#endif
+ if (err) {
/* The allocation failed: return NULL after putting the
* arenaobj back.
*/
@@ -1084,8 +769,9 @@
* Unless the optimizer reorders everything, being too smart...
*/
-static void *
-_PyObject_Malloc(void *ctx, size_t nbytes)
+#undef PyObject_Malloc
+void *
+PyObject_Malloc(size_t nbytes)
{
block *bp;
poolp pool;
@@ -1102,6 +788,17 @@
#endif
/*
+ * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
+ * Most python internals blindly use a signed Py_ssize_t to track
+ * things without checking for overflows or negatives.
+ * As size_t is unsigned, checking for nbytes < 0 is not required.
+ */
+ if (nbytes > PY_SSIZE_T_MAX) {
+ _Py_AllocatedBlocks--;
+ return NULL;
+ }
+
+ /*
* This implicitly redirects malloc(0).
*/
if ((nbytes - 1) < SMALL_REQUEST_THRESHOLD) {
@@ -1273,8 +970,10 @@
* last chance to serve the request) or when the max memory limit
* has been reached.
*/
+ if (nbytes == 0)
+ nbytes = 1;
{
- void *result = PyMem_Malloc(nbytes);
+ void *result = malloc(nbytes);
if (!result)
_Py_AllocatedBlocks--;
return result;
@@ -1283,8 +982,9 @@
/* free */
-static void
-_PyObject_Free(void *ctx, void *p)
+#undef PyObject_Free
+void
+PyObject_Free(void *p)
{
poolp pool;
block *lastfree;
@@ -1393,8 +1093,11 @@
unused_arena_objects = ao;
/* Free the entire arena. */
- _PyObject_Arena.free(_PyObject_Arena.ctx,
- (void *)ao->address, ARENA_SIZE);
+#ifdef ARENAS_USE_MMAP
+ munmap((void *)ao->address, ARENA_SIZE);
+#else
+ free((void *)ao->address);
+#endif
ao->address = 0; /* mark unassociated */
--narenas_currently_allocated;
@@ -1503,7 +1206,7 @@
redirect:
#endif
/* We didn't allocate this address. */
- PyMem_Free(p);
+ free(p);
}
/* realloc. If p is NULL, this acts like malloc(nbytes). Else if nbytes==0,
@@ -1511,8 +1214,9 @@
* return a non-NULL result.
*/
-static void *
-_PyObject_Realloc(void *ctx, void *p, size_t nbytes)
+#undef PyObject_Realloc
+void *
+PyObject_Realloc(void *p, size_t nbytes)
{
void *bp;
poolp pool;
@@ -1522,7 +1226,16 @@
#endif
if (p == NULL)
- return _PyObject_Malloc(ctx, nbytes);
+ return PyObject_Malloc(nbytes);
+
+ /*
+ * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
+ * Most python internals blindly use a signed Py_ssize_t to track
+ * things without checking for overflows or negatives.
+ * As size_t is unsigned, checking for nbytes < 0 is not required.
+ */
+ if (nbytes > PY_SSIZE_T_MAX)
+ return NULL;
#ifdef WITH_VALGRIND
/* Treat running_on_valgrind == -1 the same as 0 */
@@ -1550,10 +1263,10 @@
}
size = nbytes;
}
- bp = _PyObject_Malloc(ctx, nbytes);
+ bp = PyObject_Malloc(nbytes);
if (bp != NULL) {
memcpy(bp, p, size);
- _PyObject_Free(ctx, p);
+ PyObject_Free(p);
}
return bp;
}
@@ -1571,14 +1284,14 @@
* at p. Instead we punt: let C continue to manage this block.
*/
if (nbytes)
- return PyMem_Realloc(p, nbytes);
+ return realloc(p, nbytes);
/* C doesn't define the result of realloc(p, 0) (it may or may not
* return NULL then), but Python's docs promise that nbytes==0 never
* returns NULL. We don't pass 0 to realloc(), to avoid that endcase
* to begin with. Even then, we can't be sure that realloc() won't
* return NULL.
*/
- bp = PyMem_Realloc(p, 1);
+ bp = realloc(p, 1);
return bp ? bp : p;
}
@@ -1588,6 +1301,24 @@
/* pymalloc not enabled: Redirect the entry points to malloc. These will
* only be used by extensions that are compiled with pymalloc enabled. */
+void *
+PyObject_Malloc(size_t n)
+{
+ return PyMem_MALLOC(n);
+}
+
+void *
+PyObject_Realloc(void *p, size_t n)
+{
+ return PyMem_REALLOC(p, n);
+}
+
+void
+PyObject_Free(void *p)
+{
+ PyMem_FREE(p);
+}
+
Py_ssize_t
_Py_GetAllocatedBlocks(void)
{
@@ -1613,6 +1344,10 @@
#define DEADBYTE 0xDB /* dead (newly freed) memory */
#define FORBIDDENBYTE 0xFB /* untouchable bytes at each end of a block */
+/* We tag each block with an API ID in order to tag API violations */
+#define _PYMALLOC_MEM_ID 'm' /* the PyMem_Malloc() API */
+#define _PYMALLOC_OBJ_ID 'o' /* The PyObject_Malloc() API */
+
static size_t serialno = 0; /* incremented on each debug {m,re}alloc */
/* serialno is always incremented via calling this routine. The point is
@@ -1695,18 +1430,58 @@
p[2*S+n: 2*S+n+S]
Copies of FORBIDDENBYTE. Used to catch over- writes and reads.
p[2*S+n+S: 2*S+n+2*S]
- A serial number, incremented by 1 on each call to _PyMem_DebugMalloc
- and _PyMem_DebugRealloc.
+ A serial number, incremented by 1 on each call to _PyObject_DebugMalloc
+ and _PyObject_DebugRealloc.
This is a big-endian size_t.
If "bad memory" is detected later, the serial number gives an
excellent way to set a breakpoint on the next run, to capture the
instant at which this block was passed out.
*/
-static void *
-_PyMem_DebugMalloc(void *ctx, size_t nbytes)
+/* debug replacements for the PyMem_* memory API */
+void *
+_PyMem_DebugMalloc(size_t nbytes)
{
- debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
+ return _PyObject_DebugMallocApi(_PYMALLOC_MEM_ID, nbytes);
+}
+void *
+_PyMem_DebugRealloc(void *p, size_t nbytes)
+{
+ return _PyObject_DebugReallocApi(_PYMALLOC_MEM_ID, p, nbytes);
+}
+void
+_PyMem_DebugFree(void *p)
+{
+ _PyObject_DebugFreeApi(_PYMALLOC_MEM_ID, p);
+}
+
+/* debug replacements for the PyObject_* memory API */
+void *
+_PyObject_DebugMalloc(size_t nbytes)
+{
+ return _PyObject_DebugMallocApi(_PYMALLOC_OBJ_ID, nbytes);
+}
+void *
+_PyObject_DebugRealloc(void *p, size_t nbytes)
+{
+ return _PyObject_DebugReallocApi(_PYMALLOC_OBJ_ID, p, nbytes);
+}
+void
+_PyObject_DebugFree(void *p)
+{
+ _PyObject_DebugFreeApi(_PYMALLOC_OBJ_ID, p);
+}
+void
+_PyObject_DebugCheckAddress(const void *p)
+{
+ _PyObject_DebugCheckAddressApi(_PYMALLOC_OBJ_ID, p);
+}
+
+
+/* generic debug memory api, with an "id" to identify the API in use */
+void *
+_PyObject_DebugMallocApi(char id, size_t nbytes)
+{
uchar *p; /* base address of malloc'ed block */
uchar *tail; /* p + 2*SST + nbytes == pointer to tail pad bytes */
size_t total; /* nbytes + 4*SST */
@@ -1717,14 +1492,14 @@
/* overflow: can't represent total as a size_t */
return NULL;
- p = (uchar *)api->alloc.malloc(api->alloc.ctx, total);
+ p = (uchar *)PyObject_Malloc(total);
if (p == NULL)
return NULL;
/* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */
write_size_t(p, nbytes);
- p[SST] = (uchar)api->api_id;
- memset(p + SST + 1, FORBIDDENBYTE, SST-1);
+ p[SST] = (uchar)id;
+ memset(p + SST + 1 , FORBIDDENBYTE, SST-1);
if (nbytes > 0)
memset(p + 2*SST, CLEANBYTE, nbytes);
@@ -1742,27 +1517,25 @@
Then fills the original bytes with DEADBYTE.
Then calls the underlying free.
*/
-static void
-_PyMem_DebugFree(void *ctx, void *p)
+void
+_PyObject_DebugFreeApi(char api, void *p)
{
- debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
uchar *q = (uchar *)p - 2*SST; /* address returned from malloc */
size_t nbytes;
if (p == NULL)
return;
- _PyMem_DebugCheckAddress(api->api_id, p);
+ _PyObject_DebugCheckAddressApi(api, p);
nbytes = read_size_t(q);
nbytes += 4*SST;
if (nbytes > 0)
memset(q, DEADBYTE, nbytes);
- api->alloc.free(api->alloc.ctx, q);
+ PyObject_Free(q);
}
-static void *
-_PyMem_DebugRealloc(void *ctx, void *p, size_t nbytes)
+void *
+_PyObject_DebugReallocApi(char api, void *p, size_t nbytes)
{
- debug_alloc_api_t *api = (debug_alloc_api_t *)ctx;
uchar *q = (uchar *)p;
uchar *tail;
size_t total; /* nbytes + 4*SST */
@@ -1770,9 +1543,9 @@
int i;
if (p == NULL)
- return _PyMem_DebugMalloc(ctx, nbytes);
+ return _PyObject_DebugMallocApi(api, nbytes);
- _PyMem_DebugCheckAddress(api->api_id, p);
+ _PyObject_DebugCheckAddressApi(api, p);
bumpserialno();
original_nbytes = read_size_t(q - 2*SST);
total = nbytes + 4*SST;
@@ -1789,12 +1562,12 @@
* case we didn't get the chance to mark the old memory with DEADBYTE,
* but we live with that.
*/
- q = (uchar *)api->alloc.realloc(api->alloc.ctx, q - 2*SST, total);
+ q = (uchar *)PyObject_Realloc(q - 2*SST, total);
if (q == NULL)
return NULL;
write_size_t(q, nbytes);
- assert(q[SST] == (uchar)api->api_id);
+ assert(q[SST] == (uchar)api);
for (i = 1; i < SST; ++i)
assert(q[SST + i] == FORBIDDENBYTE);
q += 2*SST;
@@ -1816,8 +1589,8 @@
* and call Py_FatalError to kill the program.
* The API id, is also checked.
*/
-static void
-_PyMem_DebugCheckAddress(char api, const void *p)
+ void
+_PyObject_DebugCheckAddressApi(char api, const void *p)
{
const uchar *q = (const uchar *)p;
char msgbuf[64];
@@ -1869,7 +1642,7 @@
}
/* Display info to stderr about the memory block at p. */
-static void
+void
_PyObject_DebugDumpAddress(const void *p)
{
const uchar *q = (const uchar *)p;
--
Repository URL: http://hg.python.org/cpython
More information about the Python-checkins
mailing list