[Numpy-discussion] High-quality memory profiling for numpy in python 3.5 / volunteers needed

[Julian Taylor]

On 15.04.2014 18:39, Nathaniel Smith wrote:
> On Tue, Apr 15, 2014 at 4:08 PM, Julian Taylor
> <jtaylor.debian at googlemail.com <mailto:jtaylor.debian at googlemail.com>>
> wrote:
>> On Tue, Apr 15, 2014 at 3:07 PM, Nathaniel Smith <njs at pobox.com
> <mailto:njs at pobox.com>> wrote:
>>> On Tue, Apr 15, 2014 at 12:06 PM, Julian Taylor
>>> <jtaylor.debian at googlemail.com
> <mailto:jtaylor.debian at googlemail.com>> wrote:
>>>>> Good news, though! python-dev is in favor of adding calloc() to the
>>>>> core allocation interfaces, which will let numpy join the party. See
>>>>> python-dev thread:
>>>>> https://mail.python.org/pipermail/python-dev/2014-April/133985.html
>>>>>
>>>>> It would be especially nice if we could get this into 3.5, since it
>>>>> seems likely that lots of numpy users will be switching to 3.5 when it
>>>>> comes out, and having a good memory tracing infrastructure there
>>>>> waiting for them make it even more awesome.
>>>>>
>>>>> Anyone interested in picking this up?
>>>>> http://bugs.python.org/issue21233
>>>>
>>>> Hi,
>>>> I think it would be a better idea to instead of API functions for one
>>>> different type of allocator we get access to use the python hooks
>>>> directly with whatever allocator we want to use.
>>>
>>> Unfortunately, that's not how the API works. The way that third-party
>>> tracers register a 'hook' is by providing a new implementation of
>>> malloc/free/etc. So there's no general way to say "please pretend to
>>> have done a malloc".
>>>
>>> I guess we could potentially request the addition of
>>> fake_malloc/fake_free functions.
>>
>> Unfortunate, looking at the pep it seems either you have a custom
>> allocator or you have tracing but not both (unless you trace
>> yourself).
>> This seems like quite a limitation.
> 
> I don't think this is right - notice the PyMem_GetAllocator function,
> which lets you grab the old allocator. This means you can write a
> tracing "allocator" which just does its tracing and then delegates to
> the old allocator. (And looking at _tracemalloc.c this does seem to be
> how it works.) This means that any full allocator replacement has to be
> enabled first before any tracing allocator is enabled, but that's okay,
> because a full allocator has to be inserted *very* early in any case
> (like, before any allocations have happened) and can never be removed,
> so this doesn't seem so bad.
> 

Hm right, so we don't need python to provide calloc, we just rely on
tracemalloc using our custom allocator after its done its tracing.
But in this approach on loading numpy we would replace the allocator of
all modules imported in the same process and any other module that
changes the allocator would also change the allocator numpy uses.
That sounds dangerous.