Debugging memory leaks

[Frank Millman]

"writeson" <doug.farrell at gmail.com> wrote in message 
news:09917103-b35e-4728-8fea-bcb4ce2bd1af at googlegroups.com...
> Hi all,
>
> I've written a program using Twisted that uses SqlAlchemy to access a 
> database using threads.deferToThread(...) and SqlAlchemy's 
> scoped_session(...). This program runs for a long time, but leaks memory 
> slowly to the point of needing to be restarted. I don't know that the 
> SqlAlchemy/threads thing is the problem, but thought I'd make you aware of 
> it.
>
> Anyway, my real question is how to go about debugging memory leak problems 
> in Python, particularly for a long running server process written with 
> Twisted. I'm not sure how to use heapy or guppy, and objgraph doesn't tell 
> me enough to locate the problem. If anyone as any suggestions or pointers 
> it would be very much appreciated!
>
> Thanks in advance,
> Doug

You have received lots of good advice, but there is one technique that I 
have found useful that has not been mentioned.

As you are probably aware, one of the main causes of a 'memory leak' in 
python is an object that is supposed to be garbage collected, but hangs 
around because there is still a reference pointing to it.

You cannot directly confirm that an object has been deleted, because 
invoking its '__del__' method causes side-effects which can prevent it from 
being deleted even if it is otherwise ok.

However, there is an indirect way of confirming it - a 'DelWatcher' class. I 
got this idea from a thread on a similar subject in this forum a long time 
ago. Here is how it works.

class DelWatcher:
    def __init__(self, obj):
        # do not store a reference to obj - that would create a circular 
reference
        # store some attribute that uniquely identifies the 'obj' instance
        self.name = obj.name
        print(self.name, 'created')
    def __del__(self):
        print(self.name, 'deleted')

class MyClass:
    def __init__(self, ...):
        [...]
        self._del = DelWatcher(self)

Now you can watch the objects as they are created, and then check that they 
are deleted when you expect them to be.

This can help to pinpoint where the memory leak is occurring.

HTH

Frank Millman