[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

[Victor Stinner]

2013/6/15 Christian Heimes <christian at python.org>:
> Am 15.06.2013 14:22, schrieb Nick Coghlan:
>> However, it's still desirable to be able to monitor those direct
>> allocations in debug mode, thus it makes sense to have a GIL protected
>> direct allocation API as well. You could try to hide the existence of
>> the latter behaviour and treat it as a private API, but why? For
>> custom allocators, it's useful to be able to *ensure* you can bypass
>> CPython's small object allocator, rather than having to rely on it
>> being bypassed for allocations above a certain size.
>
> There is even more to it. We like to keep track of memory allocations in
> libraries that are wrapped by Python's extension modules, e.g. expat,
> openssl etc. Almost every library has a hook to set a custom memory
> allocator, either globally (CRYPTO_set_mem_functions) or for each object
> (XML_ParserCreate_MM's XML_Memory_Handling_Suite).

I just create the issue http://bugs.python.org/issue18227: "Use Python
memory allocators in external libraries like zlib or OpenSSL".

Is it possible to detect if Python is used as a standalone application
(the classic "python" program) or if Python is embedded? If it is
possible, we can modify the "global" memory allocators of a library.
Otherwise, it is more tricky. Should Python sets its "own" memory
allocators? Maybe only if PyMem_SetRawAllocators() was called?

> Eventually I would like to ban direct usage of malloc() from Python's
> core and patch all memory management through our API.

I already create issue http://bugs.python.org/issue18203 for this part.

Victor