Hunting a memory leak

[Debian User]

Hi,

I'm trying to discover a memory leak on a program of mine. I've taken
several approaches, but the leak still resists to appear.

First of all, I've tried to use the garbage collector to look for
uncollectable objects. I've used the next:

    # at the beginning of code
    gc.enable()
    gc.set_debug(gc.DEBUG_LEAK)

    <snip>

    # at the end of code:
    print "\nGARBAGE:"
    gc.collect()

    print "\nGARBAGE OBJECTS:"
    for x in gc.garbage:
        s = str(x)
        print type(x),"\n   ", s

With that, I get no garbage objects.

Then I've taken an approach that I've seen in python developers list
contributed by Walter Dörwald, that basically consists in creating a
debug version of python, create a unitest with the leaking code, and
modify the unittest.py to extract the increment of total reference
counting in that code (see
http://aspn.activestate.com/ASPN/Mail/Message/python-dev/1770868).

With that, I see that my reference count grows by one each time the
test execute. But the problem is: is there some way to look at the
object (or make a memory dump) that is leaking?.

I've used valgrind (http://developer.kde.org/~sewardj/) to see if it
could detect the leak. In fact, it detects a bunch of them, but I am
afraid that they are not related with the leak I'm looking for. I am
saying that because, when I loop over my leaky code, valgrind always
report the same amount of leaky memory, independently of the number of
iterations (while top is telling me that memory use is growing!).

My code uses extension modules in C, so I am afraid this does not
contribute to alleviate the problem. I think all the malloc are
correctly freed, but I can't be sure (however, valgrind does not
detect nothing wrong in the extension).

I am sorry, but I cannot be more explicit about the code because it
is quite complex (it is the PyTables package, http://pytables.sf.net),
and I was unable to make a simple example to be published
here. However, if anyone is tempted to have a look at the code, you
can download it from
(http://sourceforge.net/project/showfiles.php?group_id=63486). I am
attaching a unittest that exposes the leak.

I am a bit desperate. Any hint?

Francesc Alted

--

# Unittest to expose the memory leak
import sys
import unittest
import os
import tempfile

from tables import *
# Next imports are only necessary for this test suite
#from tables import Group, Leaf, Table, Array

verbose = 0

class WideTreeTestCase(unittest.TestCase):

    def test00_Leafs(self):

        import time
        maxchilds = 2
        if verbose:
            print '\n', '-=' * 30
            print "Running %s.test00_wideTree..." % \
                  self.__class__.__name__
            print "Maximum number of childs tested :", maxchilds
        # Open a new empty HDF5 file
        file = tempfile.mktemp(".h5")
        #file = "test_widetree.h5"

        fileh = openFile(file, mode = "w")
        if verbose:
            print "Children writing progress: ",
        for child in range(maxchilds):
            if verbose:
                print "%3d," % (child),
            a = [1, 1]
            fileh.createGroup(fileh.root, 'group' + str(child),
                              "child: %d" % child)
            # Comment the createArray call to see the leak disapear
            fileh.createArray("/group" + str(child), 'array' + str(child),
                              a, "child: %d" % child)
        if verbose:
            print
        # Close the file
        fileh.close()

        
#----------------------------------------------------------------------

def suite():
    theSuite = unittest.TestSuite()
    theSuite.addTest(unittest.makeSuite(WideTreeTestCase))

    return theSuite


if __name__ == '__main__':
    unittest.main(defaultTest='suite')