[Numpy-discussion] Strange memory consumption in numpy?

[Robert Kern]

On Thu, May 16, 2013 at 1:32 PM, Martin Raspaud <martin.raspaud at smhi.se> wrote:
> On 16/05/13 10:26, Robert Kern wrote:
>
>>> Can anyone give a reasonable explanation ?
>>
>> memory_profiler only looks at the amount of memory that the OS has
>> allocated to the Python process. It cannot measure the amount of
>> memory actually given to living objects. Python does not always return
>> memory back to the OS immediately when it frees the memory for an
>> object. Your two observations are linked. Python freed the memory of
>> arr2 immediately, but it did not return the memory to the OS, so
>> memory_profiler could not notice it. When arr3 is allocated, it
>> happened to fit into the block of memory that arr2 once owned, so
>> Python's memory allocator just used that block again. Since Python did
>> not have to go out to the OS to get more memory, memory_profiler could
>> not notice that, either.
>
> Robert,
>
> Thanks a lot for the clear explanation, it makes perfect sense now.
>
> You're talking about living objects, but as I understand the few memory
> profilers I found around the web for python can't track numpy arrays.
> Any pointers on something that would work with numpy ?

meliae has special support for numpy.ndarray objects. It's a little
broken, in that it will double-count views, but you can provide a
better specialization if you wish (look for the add_special_size()
function).

https://launchpad.net/meliae

--
Robert Kern