[Matrix-SIG] Freeing malloc'd array in Python

[Aaron Watters]

I might be wrong, but i think the problem is that
Python keeps a last value reference named _ (underscore);
the del a deletes the *name* a but the object formerly
named a still has another name _, maybe try this instead

a = my_range(10000)
a = None
1

Now the object should be gone.  Another thing that can
happen is the object can get stuck on a traceback object

import sys
sys.last_traceback = None

If you really have a memory leak this should cause problems

from time import sleep
for i in range(100):
     x = my_range(10000)
     print "sleeping"; sleep(2)

...watch the process size from another window.  If it really
is leaking, maybe recomplain.   -- Aaron Watters

Scott M. Ransom wrote:

> Hello,
>
> I am trying to wrap a large pulsar data analysis library written in C so
> that I can use it in Python.  I am using the Numeric extension to
> represent many vectors and matrices that I use.
>
> I am new to Python and do not (yet!) have a good grasp of Python's
> reference counting method of garbage collection -- as you will probably
> see from my code snippet below... ;)
>
> The problem that I am running into is that many of my routines allocate
> arrays using malloc() from 'C' and then return these newly created
> arrays to the user.  In my 'C' routines I simple free() the array in the
> main program when I don't need it anymore.
>
> When I wrap these functions onto Numeric arrays, though, Python does not
> seem to want to free the data that I malloc'd.  For example, using the
> code attached at the bottom of this note and compiled (on Linux) using:
>
> gcc -shared -I/usr/local/include/python1.5 -Wall -fpic -g -O my_range.c
> -o my_range.so
>
> and then imported into Python,  I get the following:
>
> Python 1.5.1 (#8, Sep  3 1998, 19:02:41)  [GCC egcs-2.90.29 980515 (egc
> on linux2
> Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
> >>> from Numeric import *
> >>> from my_range import *
> >>> from sys import *
> >>> a = my_range(10)
> >>> getrefcount(a)
> 2
> >>> a
> array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])
> >>> del(a)
> >>> getrefcount(a)
> Traceback (innermost last):
>   File "<stdin>", line 1, in ?
> NameError: a
> >>> a = my_range(100000)
> >>> del(a)
> >>>
>
> del(a) obviously deletes the PyArrayObject, but examining the memory
> usage before and after the a = my_range(100000) and the corresponding
> del(a), it is obvious that Python is not touching the actual data in the
> PyArrayObject -- a bigtime memory leak.
>
> I examined the code in arrayobject.c and arrayobject.h and my initial
> impressions are that I need to somehow manually free the data portion of
> the PyArrayObject when I am finished with it in Python -- before I del()
> the PyArrayObject.  I unfortunately have no idea how to do this.  Do I
> need another helper function in my wrapper code to free() the data in
> the PyArrayObject()?  Or am I missing something even easier?
>
> It seems that I have also been warned (from arrayobject.h) not to use
> PyArray_FromDimsAndData() to create my PyArrayObject (any ideas on how
> to use something else?)
>
> Thanks in advance for your help,
>
> Scott Ransom
>
> ---------my_range.c------------
> #include <stdlib.h>
> #include <stdio.h>
> #include "Python.h"
> #include "arrayobject.h"
>
> /* Here is an example library function that returns an array (a 1D
>    vector).  This is just representative -- the actual functions in my
>    library are much more complicated.  They are all malloc'd,
>    though. */
>
> double * my_range(long n)
> {
>   double *v;
>   long i;
>
>   v = (double *) malloc((size_t) (sizeof(double) * n));
>   if (!v) {
>     printf("\nAllocation error in my_range()\n");
>     exit(1);
>   }
>   for (i=0; i<n; i++) v[i] = (double) i;
>   return v;
> }
>
> /* Here is a representative wrapper function that I am using to
>    interface the above routine with Python (using Numeric arrays). */
>
> static PyObject *wrap_my_range(PyObject *self, PyObject *args)
> {
>   PyObject *obj;
>   PyArrayObject *arr;
>   double *result;
>   long n;
>
>   self = self;
>   if(!PyArg_ParseTuple(args,"O:wrap_my_range",&obj))
>     return NULL;
>   n = PyInt_AsLong((PyObject *)obj);
>   result = my_range(n);
>   arr = (PyArrayObject *)PyArray_FromDimsAndData(1, (int *)&n,
> PyArray_DOUBLE, \
>                                                  (char *) result);
>   if (arr == NULL) return NULL;
>   return (PyObject *)arr;
> }
>
> /* Here are the module initialization functions */
>
> static PyObject *ErrorObject;
> static PyMethodDef my_range_method[] = {
>   { "my_range", wrap_my_range, 1 },
>   { NULL, NULL }
> };
>
> void initmy_range()
> {
>   PyObject *m, *d;
>
>   m = Py_InitModule("my_range", my_range_method);
>   d = PyModule_GetDict(m);
>   import_array();
>   ErrorObject = PyString_FromString("my_range.error");
>   PyDict_SetItemString(d, "error", ErrorObject);
>   if (PyErr_Occurred())
>     Py_FatalError("can't initialize module my_range");
> }
>
> --
> Scott M. Ransom
> Phone:  (580) 536-7215             Address:  703 SW Chaucer Cir.
> email:  ransom@cfa.harvard.edu               Lawton, OK  73505
> PGP Fingerprint: D2 0E D0 10 CD 95 06 DA  EF 78 FE 2B CB 3A D3 53
>
> _______________________________________________
> Matrix-SIG maillist  -  Matrix-SIG@python.org
> http://www.python.org/mailman/listinfo/matrix-sig