Memory leak in TableIO?

[John Machin]

Mark Fardal <fardal at corvus.phys.uvic.ca> wrote in message news:<yj1hecb53d1.fsf at corvus.phys.uvic.ca>...
> Hi,
> 
> I use Mike Miller's TableIO to read ascii tables, since it's very
> fast.  However, I have noticed that when I use it repeatedly to read
> large data files, it soaks up more and more memory (as shown with top)
> and eventually crashes.  This is using TableIO v 1.7, Python 2.2 on
> Red Hat linux.  Has anyone else had this problem or know of a solution?  
> I glanced over the code in _tableio.c and didn't see anything
> obviously wrong, but C extensions are probably beyond my competence.
> 
> thanks,
> Mark Fardal

At first glance, there are several instances of

status = PyList_Append( myList, PyFloat_FromDouble(atof(string)) );

This needs to be changed to

tmp_obj = PyFloat_FromDouble(atof(string)) );
status = PyList_Append( myList, tmp_obj );
Py_DECREF(tmp_obj);

There may well be other leaks, there may well be other problems, but
plugging the above leak of one Python float object per table element
should get you a lot further ...

You will notice that there is no error checking on the result of the
PyFloat_FromDouble() call, presumably because atof() is expected to
return a legal double even if the input string is not a valid
representation of a floating-point number [see example below], and so
the presumption is that PyFloat_FromDouble() cannot fail. However
given your opening remark that you use this module "since it's very
fast", this may not be a concern to you :-)

..>cat junk.txt
123 xyz 789

..>python
>>> import _tableio
>>> _tableio.readTable("junk.txt","#")
{'rows': 1, 'data': [123.0, 0.0, 789.0], 'columns': 3, 'filename':
'junk.txt'}