[Python-checkins] peps: PEP 445

[victor.stinner]

http://hg.python.org/peps/rev/0971c633e30b
changeset:   4939:0971c633e30b
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue Jun 18 02:46:10 2013 +0200
summary:
  PEP 445

files:
  pep-0445.txt |  70 +++++++++++++++++++++++++--------------
  1 files changed, 45 insertions(+), 25 deletions(-)


diff --git a/pep-0445.txt b/pep-0445.txt
--- a/pep-0445.txt
+++ b/pep-0445.txt
@@ -12,7 +12,7 @@
 Abstract
 ========
 
-Add new APIs to customize memory allocators
+Add new APIs to customize memory allocators.
 
 
 Rationale
@@ -20,15 +20,19 @@
 
 Use cases:
 
-* Application embedding Python wanting to use a custom memory allocator
-  to allocate all Python memory somewhere else or with a different algorithm
+* Application embedding Python may want to isolate Python memory from the
+  memory of the application, or may want to different memory allocator
+  optimized for its Python usage
 * Python running on embedded devices with low memory and slow CPU.
   A custom memory allocator may be required to use efficiently the memory
   and/or to be able to use all memory of the device.
-* Debug tool to track memory leaks
-* Debug tool to detect buffer underflow, buffer overflow and misuse
-  of Python allocator APIs
-* Debug tool to inject bugs, simulate out of memory for example
+* Debug tool to:
+
+  - track memory leaks
+  - get the Python filename and line number where an object was allocated
+  - detect buffer underflow, buffer overflow and detect misuse of Python
+    allocator APIs (builtin Python debug hooks)
+  - force allocation to fail to test handling of ``MemoryError`` exception
 
 API:
 
@@ -62,8 +66,8 @@
   - ``void _PyObject_GetArenaAllocators(void **ctx_p, void* (**malloc_p) (void *ctx, size_t size), void (**free_p) (void *ctx, void *ptr, size_t size))``
   - ``void _PyObject_SetArenaAllocators(void *ctx, void* (*malloc) (void *ctx, size_t size), void (*free) (void *ctx, void *ptr, size_t size))``
 
-* Add a new function to setup debug hooks after memory allocators were
-  replaced:
+* Add a new function to setup Python builtin debug hooks when memory
+  allocators are replaced:
 
   - ``void PyMem_SetupDebugHooks(void)``
 
@@ -71,19 +75,19 @@
 Use these new APIs
 ------------------
 
-* ``PyMem_Malloc()`` and ``PyMem_Realloc()`` now always call ``malloc()`` and
+* ``PyMem_Malloc()`` and ``PyMem_Realloc()`` always call ``malloc()`` and
   ``realloc()``, instead of calling ``PyObject_Malloc()`` and
   ``PyObject_Realloc()`` in debug mode
 
-* ``PyObject_Malloc()`` now falls back on ``PyMem_Malloc()`` instead of
-  ``malloc()`` if size is bigger than ``SMALL_REQUEST_THRESHOLD``, and
-  ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()`` instead of
-  ``realloc()``
+* ``PyObject_Malloc()`` falls back on ``PyMem_Malloc()`` instead of
+  ``malloc()`` if size is greater or equal than ``SMALL_REQUEST_THRESHOLD``
+  (512 bytes), and ``PyObject_Realloc()`` falls back on ``PyMem_Realloc()``
+  instead of ``realloc()``
 
 * Replace direct calls to ``malloc()`` with ``PyMem_Malloc()``, or
   ``PyMem_RawMalloc()`` if the GIL is not held
 
-* Configure external libraries like zlib or OpenSSL to use
+* Configure external libraries like zlib or OpenSSL to allocate memory using
   ``PyMem_RawMalloc()``
 
 
@@ -93,7 +97,8 @@
 Use case 1: Replace Memory Allocators, keep pymalloc
 ----------------------------------------------------
 
-Setup your custom memory allocators, keeping pymalloc::
+Setup your custom memory allocators, keeping pymalloc. Dummy example wasting 2
+bytes per allocation, and 10 bytes per arena::
 
     #include <stdlib.h>
 
@@ -156,7 +161,9 @@
 
 If your allocator is optimized for allocation of small objects (less than 512
 bytes) with a short liftime, you can replace override pymalloc (replace
-``PyObject_Malloc()``).  Example::
+``PyObject_Malloc()``).
+
+Dummy Example wasting 2 bytes per allocation::
 
     #include <stdlib.h>
 
@@ -203,7 +210,7 @@
 Use case 3: Setup Allocator Hooks
 ---------------------------------
 
-Setup hooks on memory allocators::
+Example to setup hooks on memory allocators::
 
     struct {
         PyMemAllocators pymem;
@@ -314,13 +321,20 @@
 * ``PYALLOC_PYOBJECT``
 
 
-Setup Builtin Debug Hooks
--------------------------
+Add a new PYDEBUGMALLOC environment variable
+--------------------------------------------
 
-To be able to use Python debug functions (like ``_PyMem_DebugMalloc()``) even
-when a custom memory allocator is set, an environment variable
-``PYDEBUGMALLOC`` can be added to set these debug function hooks, instead of
-the new function ``PyMem_SetupDebugHooks()``.
+To be able to use Python builtin debug hooks even when a custom memory
+allocator is set, an environment variable ``PYDEBUGMALLOC`` can be added to
+setup these debug function hooks, instead of adding the new function
+``PyMem_SetupDebugHooks()``. If the environment variable is present,
+``PyMem_SetRawAllocators()``, ``PyMem_SetAllocators()`` and
+``PyObject_SetAllocators()`` will reinstall automatically the hook on top of
+the new allocator.
+
+An new environment variable would make the Python initialization even more
+complex. The `PEP 432 <http://www.python.org/dev/peps/pep-0432/>`_ tries to
+simply the CPython startup sequence.
 
 
 Use macros to get customizable allocators
@@ -336,6 +350,12 @@
 Use C macros using ``__FILE__`` and ``__LINE__`` to get the C filename
 and line number of a memory allocation.
 
+Passing a filename and a line number to each allocator makes the API more
+complex: pass 3 new arguments instead of just a context argument, to each
+allocator function. GC allocator functions should also be patched,
+``_PyObject_GC_Malloc()`` is used in many C functions for example. Such changes
+add too much complexity, for a little gain.
+
 
 No context argument
 -------------------
@@ -446,7 +466,7 @@
   <http://bugs.python.org/issue13483>`_
 * `Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
   isn't thread safe <http://bugs.python.org/issue16742>`_
-* `Issue #18203: Replace calls to malloc() with PyMem_Malloc()
+* `Issue #18203: Replace calls to malloc() with PyMem_Malloc() or PyMem_RawMalloc()
   <http://bugs.python.org/issue18203>`_
 * `Issue #18227: Use Python memory allocators in external libraries like zlib
   or OpenSSL <http://bugs.python.org/issue18227>`_

-- 
Repository URL: http://hg.python.org/peps