[SciPy-user] Weave Memory Leak?

[eric jones]

Hey Greg,

It is a ref count issue.  If you change your code at the end of your 
function [ly.py:299 or so] from:

    return_val = (PyObject *) ret_array;

to:

    return_val = (PyObject *) ret_array;
    Py_XDECREF((PyObject*)ret_array);

you'll be set.  The problem is that PyArray_FromDims creates a PyObject* 
with refcount 1 and then assigning this to the py::object return_val 
also increments the refcount by 1 because it now refers to the same 
object.  return_val now has a refcount of 2.  Before returning, you need 
to release "your" reference to ret_array by calling the Py_XDECREF().  
This will set the refcount back to 1 which is the desired value before 
returning.

Refcounts are tricky and sometimes hard to get right.  Weave tries hard 
but doesn't get rid of reference management issues in all possible cases. 

I guess the best rule here is that assigning a PyObject* to return_val 
always increases its refcount.  If you also hold a refcount you must 
release it. 

When assigning non-python objects (int, float, etc.) the correct type of 
python object is created on the fly and therefore has a single reference 
count.

One other approach is to never create raw PyObject* objects within the 
function.  This way you don't have to mess with ref counts at all.  If a 
weave function returns an output array that has a known size, I'll 
typically create it in python and then pass it in as one of the weave 
arguments.  It isn't quite as elegant as creating it internally, but 
then noone is going to look at the following and complain about the 
elegance of an extra calling argument. :-)

    rval = weave.inline(thecode, ['yi', 'ts', 'params', 'hinit', 'hmin', 
'hmax', 'tol', 'verbose'],
                    support_code=eqns['code'],
                    sources=['nr-integrate.cpp', 'nrutil.cpp'],
                    include_dirs=[os.getcwd()],
                    headers=['"nrutil.h"', '"nr-integrate.h"'],
                    force=recompile)

In your case, you can pre-allocate ret_array and then pass it into the 
function. 

Someday, it'd also be nice to include a py::array object in weave that 
handles the refcounting during assignment transparently (but not today...).

hope that helps,
eric

Greg Novak wrote:

>A few days ago I wrote the list about a memory leak in code involving
>Weave.  Now I'm ready to run it up the flagpole.
>
>I've put the code here:
>http://www.ucolick.org/~novak/memory-leak.tgz
>
>The problem is in the integrate() python function.  If I replace it
>with code that doesn't use Weave (just returns reasonable fake data),
>there's no memory problem.
> 
>If anyone is kind enough to look at the code, note how the call to
>odeint() within integrate() is commented out.  So all I do is allocate
>some space w/ malloc through dmatrix(), fill it with fake data,
>allocate a new Numeric array, copy the fake data into it, deallocate
>the space allocated w/ malloc, and then set return_val to the Numeric
>array.  There's no way to return from the function without calling
>dmatrix_free().
>
>I thought at first that the reference count to the aforementioned
>Numeric array was incremented one too many times, but if I call
>Py_DECREF() on it, I get a core dump, presumably b/c the array is
>de-allocated.
>
>That's all.  Thanks a bunch for any assistance.
>
>Greg
>
>_______________________________________________
>SciPy-user mailing list
>SciPy-user at scipy.net
>http://www.scipy.net/mailman/listinfo/scipy-user
>  
>