[SciPy-Dev] memory leak in LinearNDInterpolator?

[Ralf Gommers]

On Wed, Nov 21, 2018 at 3:19 PM Aronne Merrelli <aronne.merrelli at gmail.com>
wrote:

> Hello,
>
> I am having a problem with the LinearNDInterpolator in SciPy, where it
> looks like there is a memory leak. I have a simple example here, all I do
> is setup a very small LinearNDInterpolator object and call the interpolator
> many times in a loop:
>
> https://gist.github.com/aronnem/707dfeec07b74888fb7390d828684184
>
> Note that if the "pts_list" and "vals" are swapped with the integer
> versions, there is no leak. Using the floats seems to produce the leak. I
> have tried both conda and Canopy installed environments, they all seem to
> have the same behavior. For example, here is the output I get with a scipy
> 1.1.0 compiled with Cython 0.28.5, numpy 1.13.3 on Mac OS X:
>

I can reproduce that as well with numpy 1.14.3, cython 0.29.0, scipy master.


> rss_used=         4 KB vms_used=         0 KB
> rss_used=      2516 KB vms_used=      2560 KB
> rss_used=      4836 KB vms_used=      5188 KB
> rss_used=      6880 KB vms_used=      6664 KB
> rss_used=      9408 KB vms_used=     10248 KB
> rss_used=     11680 KB vms_used=     12076 KB
> rss_used=     14356 KB vms_used=     15872 KB
> rss_used=     16736 KB vms_used=     17792 KB
> rss_used=     19416 KB vms_used=     20496 KB
> rss_used=     22100 KB vms_used=     23472 KB
>
>
> The only workaround I have found so far is to downgrade my numpy version
> to 1.11.3. So that seems to suggest it is a numpy problem; however this
> function is implemented in cython, so I am not sure how to tell whether
> this is a numpy, scipy, or cython problem.
>

It does look like a real memory leak, not the memory allocator holding on
to allocations for longer than it has to. Given that numpy 1.11.3 didn't
show this behavior, it would be helpful to use git bisect on the range numy
v1.11.3-v1.13.3 to find when the numpy behavior changed. I just checked
that v1.11.3 still builds with Cython 0.29.0, so I would suggest cython
0.29.0 and scipy master.

Depending on your OS, you may need this small patch for older numpy:

diff --git a/numpy/core/src/multiarray/numpyos.c
b/numpy/core/src/multiarray/numpyos.c
index 450ec40b6..ec1e5bc2c 100644
--- a/numpy/core/src/multiarray/numpyos.c
+++ b/numpy/core/src/multiarray/numpyos.c
@@ -15,7 +15,9 @@

 #ifdef HAVE_STRTOLD_L
 #include <stdlib.h>
-#include <xlocale.h>
+#ifdef HAVE_XLOCALE_H
+    #include <xlocale.h>
+#endif
 #endif

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20181124/71673aff/attachment.html>