[Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

[Nick Coghlan]

On 20 June 2013 15:37, Victor Stinner <victor.stinner at gmail.com> wrote:
> Le jeudi 20 juin 2013, Nick Coghlan a écrit :
>>
>> > Is PyMemMappingAllocator complete enough for your usage at CCP Games?
>>
>> Can we go back to calling this the "Arena" allocator? Or at least
>> "Mapped"? When I see "Mapping" in the context of Python I think of the
>> container API, not a memory allocation API.
>
> This function is written to be able to use mmap() and VirtualAlloc(). There
> is no Python function to use directly this allocator yet, but I chose
> "memory mapping" name because it is very different than the heap and it may
> be useful for other functions than pymalloc.
>
> If I change the name, it would be called PyObject_SetArenaAllocator() with a
> PyObjectArenaAllocator structure. I'm not sure that PyMemMappingAllocator
> API is future-proof, so I'm fine to call it "arena" again.

Yeah, I think making that API specifically about pymalloc is a good
idea. It also makes it clearer that if you're bypassing pymalloc
entirely (by replacing the
object allocators), then you shouldn't need to worry about those.

>> > I hope that the PEP 445 is flexible enough to allow you to decide
>> > which functions are hooked and replaced, and which functions will be
>> > leaved unchanged. That's why I'm not in favor of the "Make
>> > PyMem_Malloc() reuse PyMem_RawMalloc() by default" alternative.
>>
>> It's also why I'm in favour of the "domain" API rather than separate
>> functions.
>>
>> 1. In the initial iteration, just have the three basic domains (raw,
>> interpreter, objects). Replacing allocators for third party libraries is the
>> responsibility of embedding applications.
>>
>> 2. In a later iteration, add "PyMem_AddDomain" and "PyMem_GetDomains" APIs
>> so that extension modules can register new domains for wrapped libraries.
>> Replacing allocators is still the responsibility of embedding applications,
>> but there's a consistent API to do it.
>>
>> (Alternatively, we could do both now)
>
> How would you use an allocator of a new domain? PyMemBlockAllocator
> structure is not convinient, and if Py_GetAllocator() only once, you may
> loose a hook installed later.

Say that, for the current PEP, we assume we configure standard library
extension modules to use the PyMem or PyMem_Raw APIs (depending on GIL
usage), thus allowing those to be redirected automatically to an
externally configured allocator when the new PEP 445 APIs are used.

The notion I had an mind as a possible future change is that extension
modules could register a "set_allocator" callback with the interpreter
so they will be automatically notified if the allocator they're
interested in changes. However, I also realised that this would
actually be independent of the APIs in the current PEP. You could do
something like:

    typedef void (*PyMem_AllocatorSetter)(PyMemAllocator *allocator);

    void
    PyMem_AddExternalAllocator(
        PyMemAllocatorDomain domain,
        PyMemAllocatorSetter set_allocator
    );

Then, whenever the allocator for the specified domain was changed, the
"set_allocator" callback would be invoked to set the allocator in the
extension module as well. The setter would also be called immediately
on registration, using the currently defined allocator.

We don't have to do this right away (and we should give the basic API
a chance to establish itself first). I just like it as something that
the "memory domain" model may allow us to pursue in the future. (That
said, we may end up wanting something like this internally anyway,
even just for the standard library extension modules that do memory
allocations)

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia