[SciPy-Dev] memory leak in LinearNDInterpolator?

[Aronne Merrelli]

On Sun, Nov 25, 2018 at 1:32 PM Charles R Harris <charlesr.harris at gmail.com>
wrote:

>
>
> On Sun, Nov 25, 2018 at 12:10 PM Aronne Merrelli <
> aronne.merrelli at gmail.com> wrote:
>
>>
>>
>> On Sat, Nov 24, 2018 at 1:20 PM Ralf Gommers <ralf.gommers at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli <
>>> aronne.merrelli at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am having a problem with the LinearNDInterpolator in SciPy, where it
>>>> looks like there is a memory leak. I have a simple example here, all I do
>>>> is setup a very small LinearNDInterpolator object and call the interpolator
>>>> many times in a loop:
>>>>
>>>> https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
>>>>
>>>> Note that if the "pts_list" and "vals" are swapped with the integer
>>>> versions, there is no leak. Using the floats seems to produce the leak. I
>>>> have tried both conda and Canopy installed environments, they all seem to
>>>> have the same behavior. For example, here is the output I get with a scipy
>>>> 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
>>>>
>>>
>>> I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy
>>> master.
>>>
>>>
>>>> rss_used=         4 KB vms_used=         0 KB
>>>> rss_used=      2516 KB vms_used=      2560 KB
>>>> rss_used=      4836 KB vms_used=      5188 KB
>>>> rss_used=      6880 KB vms_used=      6664 KB
>>>> rss_used=      9408 KB vms_used=     10248 KB
>>>> rss_used=     11680 KB vms_used=     12076 KB
>>>> rss_used=     14356 KB vms_used=     15872 KB
>>>> rss_used=     16736 KB vms_used=     17792 KB
>>>> rss_used=     19416 KB vms_used=     20496 KB
>>>> rss_used=     22100 KB vms_used=     23472 KB
>>>>
>>>>
>>>> The only workaround I have found so far is to downgrade my numpy
>>>> version to 1.11.3. So that seems to suggest it is a numpy problem; however
>>>> this function is implemented in cython, so I am not sure how to tell
>>>> whether this is a numpy, scipy, or cython problem.
>>>>
>>>
>>> It does look like a real memory leak, not the memory allocator holding
>>> on to allocations for longer than it has to. Given that numpy 1.11.3 didn't
>>> show this behavior, it would be helpful to use git bisect on the range numy
>>> v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked
>>> that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython
>>> 0.29.0 and scipy master.
>>>
>>>
>>
>> Hi - Thank you both for confirming.
>>
>> I did manage to use bisect. All versions compiled without needing Ralf's
>> small patch. The actual bisected commits aren't really relevant, because it
>> looks like the behavior is controlled by value of the NPY_RELAXED_STRIDES_CHECKING
>> variable. This environment variable changed to default to 1/True in numpy
>> 1.12.1, if it was not set in the shell. In earlier versions, it defaulted
>> to 0/False if not set in the shell.
>>
>> It also looks like I can prevent the memory leak behavior in newer numpy
>> (at least, I checked 1.12.1 and 1.15.4), if I rebuild numpy
>> NPY_RELAXED_STRIDES_CHECKING=0.
>>
>> I do not understand what that variable is controlling, so I am still not
>> sure if this is numpy, scipy, or cython issue. Anyone have a suggestion?
>>
>> Otherwise, at least I've found a relatively simple workaround!
>>
>> thanks
>> Aronne
>>
>
> What happens if you set  `NPY_RELAXED_STRIDES_DEBUG=1` with the problem
> compile?
>
> Chuck
>

Hi -

I do not see any change with the DEBUG flag enabled.
This is with: Python 3.7.1, numpy 1.15.4, scipy 1.1.0, CentOS 6.1.


> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20181125/b9b8bf65/attachment-0001.html>