[Numpy-discussion] Solving a memory leak in a numpy extension; PyArray_ContiguousFromObject

[Dan S]

Thanks all for responses. Continuation below:

2009/4/18 Charles R Harris <charlesr.harris at gmail.com>:
>
>
> On Fri, Apr 17, 2009 at 9:25 AM, Dan S <dan.s.towell+numpy at gmail.com> wrote:
>>
>> Hi -
>>
>> I have written a numpy extension which works fine but has a memory
>> leak. It takes a single array argument and returns a single scalar.
>> After reducing the code down in order to chase the problem, I have the
>> following:
>>
>> static PyObject * kdpee_pycall(PyObject *self, PyObject *args)
>> {
>>        PyObject *input;
>>        PyArrayObject *array;
>>        int n, numdims;
>>
>>        if (!PyArg_ParseTuple(args, "O", &input))
>>                return NULL;
>>
>>        // Ensure we have contiguous, 2D, floating-point data:
>>        array = (PyArrayObject*) PyArray_ContiguousFromObject(input,
>> PyArray_DOUBLE, 2, 2);
>>
>>        if(array==NULL){
>>                printf("kdpee_py: nullness!\n");
>>                return NULL;
>>        }
>>        PyArray_XDECREF(array);
>>        Py_DECREF(array); // destroy the contig array
>>        Py_DECREF(input); // is this needed? doc says no, but it seems to fix
>            ^^^^^^^^^^^^^^^^^^^^
>
> Shouldn't be, but there might be a bug somewhere which causes the reference
> count of input to be double incremented. Does the reference count in the
> test script increase without this line?

Yes - if I comment out Py_DECREF(input), then sys.getrefcount(a) goes
from 2, to 5002, to 10002, every time I run that little 5000-fold test
iteration.

(Py_DECREF(input) is not suggested in the docs - I tried it in
desperation and for some reason it seems to reduce the problem, since
it gets rid of the refleak, although it doesn't solve the memleak.)


>>        return PyFloat_FromDouble(3.1415927); // temporary
>> }
>>
>>
>> The test code is
>>
>>      from numpy import *
>>      from kdpee import *
>>      import sys
>>      a = array( [[1,2,3,4,5,1,2,3,4,5], [6,7,8,9,0,6,7,8,9,0]])
>>      sys.getrefcount(a)
>>      for i in range(5000): tmp = kdpee(a)
>>
>>      sys.getrefcount(a)
>>
>> Every time I run this code I get "2" back from both calls to
>> sys.getrefcount(a), but I'm still getting a memory leak of about 0.4
>> MB. What am I doing wrong?
>
> So about 80 bytes/iteration? How accurate is that .4 MB? does it change with
> the size of the input array?

OK, pinning it down more precisely I get around 86--90 bytes per
iteration (using ps, which gives me Kb resolution). It does not change
if I double or quadruple the size of the input array.

Today I tried using "heapy": using the heap() and heapu() methods, I
find no evidence of anything suspicious - there is no garbage left
lying on the heap of python objects. So it's presumably something
deeper. But as you can see, my C code doesn't perform any malloc() or
suchlike, so I'm stumped.

I'd be grateful for any further thoughts.

Thanks
Dan