[SciPy-Dev] memory leak in LinearNDInterpolator?

[Charles R Harris]

On Sun, Nov 25, 2018 at 12:10 PM Aronne Merrelli <aronne.merrelli at gmail.com>
wrote:

>
>
> On Sat, Nov 24, 2018 at 1:20 PM Ralf Gommers <ralf.gommers at gmail.com>
> wrote:
>
>>
>>
>> On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli <
>> aronne.merrelli at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am having a problem with the LinearNDInterpolator in SciPy, where it
>>> looks like there is a memory leak. I have a simple example here, all I do
>>> is setup a very small LinearNDInterpolator object and call the interpolator
>>> many times in a loop:
>>>
>>> https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
>>>
>>> Note that if the "pts_list" and "vals" are swapped with the integer
>>> versions, there is no leak. Using the floats seems to produce the leak. I
>>> have tried both conda and Canopy installed environments, they all seem to
>>> have the same behavior. For example, here is the output I get with a scipy
>>> 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
>>>
>>
>> I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy
>> master.
>>
>>
>>> rss_used=         4 KB vms_used=         0 KB
>>> rss_used=      2516 KB vms_used=      2560 KB
>>> rss_used=      4836 KB vms_used=      5188 KB
>>> rss_used=      6880 KB vms_used=      6664 KB
>>> rss_used=      9408 KB vms_used=     10248 KB
>>> rss_used=     11680 KB vms_used=     12076 KB
>>> rss_used=     14356 KB vms_used=     15872 KB
>>> rss_used=     16736 KB vms_used=     17792 KB
>>> rss_used=     19416 KB vms_used=     20496 KB
>>> rss_used=     22100 KB vms_used=     23472 KB
>>>
>>>
>>> The only workaround I have found so far is to downgrade my numpy version
>>> to 1.11.3. So that seems to suggest it is a numpy problem; however this
>>> function is implemented in cython, so I am not sure how to tell whether
>>> this is a numpy, scipy, or cython problem.
>>>
>>
>> It does look like a real memory leak, not the memory allocator holding on
>> to allocations for longer than it has to. Given that numpy 1.11.3 didn't
>> show this behavior, it would be helpful to use git bisect on the range numy
>> v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked
>> that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython
>> 0.29.0 and scipy master.
>>
>>
>
> Hi - Thank you both for confirming.
>
> I did manage to use bisect. All versions compiled without needing Ralf's
> small patch. The actual bisected commits aren't really relevant, because it
> looks like the behavior is controlled by value of the NPY_RELAXED_STRIDES_CHECKING
> variable. This environment variable changed to default to 1/True in numpy
> 1.12.1, if it was not set in the shell. In earlier versions, it defaulted
> to 0/False if not set in the shell.
>
> It also looks like I can prevent the memory leak behavior in newer numpy
> (at least, I checked 1.12.1 and 1.15.4), if I rebuild numpy
> NPY_RELAXED_STRIDES_CHECKING=0.
>
> I do not understand what that variable is controlling, so I am still not
> sure if this is numpy, scipy, or cython issue. Anyone have a suggestion?
>
> Otherwise, at least I've found a relatively simple workaround!
>
> thanks
> Aronne
>

What happens if you set  `NPY_RELAXED_STRIDES_DEBUG=1` with the problem
compile?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20181125/16e10841/attachment.html>