[SciPy-Dev] memory leak in LinearNDInterpolator?

[Aronne Merrelli]

On Sat, Nov 24, 2018 at 1:20 PM Ralf Gommers <ralf.gommers at gmail.com> wrote:

>
>
> On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli <aronne.merrelli at gmail.com>
> wrote:
>
>> Hello,
>>
>> I am having a problem with the LinearNDInterpolator in SciPy, where it
>> looks like there is a memory leak. I have a simple example here, all I do
>> is setup a very small LinearNDInterpolator object and call the interpolator
>> many times in a loop:
>>
>> https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
>>
>> Note that if the "pts_list" and "vals" are swapped with the integer
>> versions, there is no leak. Using the floats seems to produce the leak. I
>> have tried both conda and Canopy installed environments, they all seem to
>> have the same behavior. For example, here is the output I get with a scipy
>> 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
>>
>
> I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy
> master.
>
>
>> rss_used=         4 KB vms_used=         0 KB
>> rss_used=      2516 KB vms_used=      2560 KB
>> rss_used=      4836 KB vms_used=      5188 KB
>> rss_used=      6880 KB vms_used=      6664 KB
>> rss_used=      9408 KB vms_used=     10248 KB
>> rss_used=     11680 KB vms_used=     12076 KB
>> rss_used=     14356 KB vms_used=     15872 KB
>> rss_used=     16736 KB vms_used=     17792 KB
>> rss_used=     19416 KB vms_used=     20496 KB
>> rss_used=     22100 KB vms_used=     23472 KB
>>
>>
>> The only workaround I have found so far is to downgrade my numpy version
>> to 1.11.3. So that seems to suggest it is a numpy problem; however this
>> function is implemented in cython, so I am not sure how to tell whether
>> this is a numpy, scipy, or cython problem.
>>
>
> It does look like a real memory leak, not the memory allocator holding on
> to allocations for longer than it has to. Given that numpy 1.11.3 didn't
> show this behavior, it would be helpful to use git bisect on the range numy
> v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked
> that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython
> 0.29.0 and scipy master.
>
>

Hi - Thank you both for confirming.

I did manage to use bisect. All versions compiled without needing Ralf's
small patch. The actual bisected commits aren't really relevant, because it
looks like the behavior is controlled by value of the
NPY_RELAXED_STRIDES_CHECKING
variable. This environment variable changed to default to 1/True in numpy
1.12.1, if it was not set in the shell. In earlier versions, it defaulted
to 0/False if not set in the shell.

It also looks like I can prevent the memory leak behavior in newer numpy
(at least, I checked 1.12.1 and 1.15.4), if I rebuild numpy
NPY_RELAXED_STRIDES_CHECKING=0.

I do not understand what that variable is controlling, so I am still not
sure if this is numpy, scipy, or cython issue. Anyone have a suggestion?

Otherwise, at least I've found a relatively simple workaround!

thanks
Aronne



> Depending on your OS, you may need this small patch for older numpy:
>
> diff --git a/numpy/core/src/multiarray/numpyos.c
> b/numpy/core/src/multiarray/numpyos.c
> index 450ec40b6..ec1e5bc2c 100644
> --- a/numpy/core/src/multiarray/numpyos.c
> +++ b/numpy/core/src/multiarray/numpyos.c
> @@ -15,7 +15,9 @@
>
>  #ifdef HAVE_STRTOLD_L
>  #include <stdlib.h>
> -#include <xlocale.h>
> +#ifdef HAVE_XLOCALE_H
> +    #include <xlocale.h>
> +#endif
>  #endif
>
> Cheers,
> Ralf
>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at python.org
> https://mail.python.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20181125/fa67d403/attachment.html>