[Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

[Nick Coghlan]

On 16 June 2013 19:50, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 16 Jun 2013 01:48:06 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
>
>> 2013/6/15 Christian Heimes <christian at python.org>:
>> > Am 15.06.2013 14:22, schrieb Nick Coghlan:
>> >> However, it's still desirable to be able to monitor those direct
>> >> allocations in debug mode, thus it makes sense to have a GIL protected
>> >> direct allocation API as well. You could try to hide the existence of
>> >> the latter behaviour and treat it as a private API, but why? For
>> >> custom allocators, it's useful to be able to *ensure* you can bypass
>> >> CPython's small object allocator, rather than having to rely on it
>> >> being bypassed for allocations above a certain size.
>> >
>> > There is even more to it. We like to keep track of memory allocations in
>> > libraries that are wrapped by Python's extension modules, e.g. expat,
>> > openssl etc. Almost every library has a hook to set a custom memory
>> > allocator, either globally (CRYPTO_set_mem_functions) or for each object
>> > (XML_ParserCreate_MM's XML_Memory_Handling_Suite).
>>
>> I just create the issue http://bugs.python.org/issue18227: "Use Python
>> memory allocators in external libraries like zlib or OpenSSL".
>>
>> Is it possible to detect if Python is used as a standalone application
>> (the classic "python" program) or if Python is embedded? If it is
>> possible, we can modify the "global" memory allocators of a library.
>
> The question is why you want to do so, not how/whether to do it.

I don't think we should be doing that ourselves - it's up to system
integrators/embedders to configure those libraries if they want to, we
shouldn't be doing it on their behalf.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia