Debugging memory leaks

[dieter]

writeson <doug.farrell at gmail.com> writes:

> ...
> Anyway, my real question is how to go about debugging memory leak problems in Python, particularly for a long running server process written with Twisted. I'm not sure how to use heapy or guppy, and objgraph doesn't tell me enough to locate the problem.

Analysing memory leaks is really difficult: huge amounts of data
is involved and usually, it is almost impossible to determine which
of the mentioned objects are leaked and which are rightfully in use.
In addition, long running Python processes usually have degrading
memory use - due to memory fragmentation. There is nothing you
can do against this.

Therefore: if the leak seems to be small, it may be much more
advicable to restart your process periodically (during times
where a restart does not hurt much) rather than try to find
(and fix) the leaks. Only when the leak is large enough that
it would force you to too frequent restarts, a deeper
analysis may be advicable (large leaks are easier to locate as well).


I have analysed memory leaks several times for Zope applications.

Zope helped me much by its "Top Refcount" functionality.
This uses the fact that a class/type instance (in many cases)
holds a reference to the corresponding class/type instance
(it seems not to work for all elementary types).
Thus looking at the refcount of the class/type gives you
an indication how many instances of this class/type are around.
Zope presents this information sorted by number.
Then you send requests against Zope and reexamine the information:
You get something like:

 Class                  June 13, 2013 8:18 am 	June 13, 2013 8:18 am 	Delta
...ApplicationManager 	15                      22                      +7
...DebugManager 	9                       12                      +3 

In the case above, my requests have created 7 additional
"ApplicationManager" and 3 additional "DebugManager" instances.

If Zope objects for which this functionality works leak,
then this is a powerful tool to detect those object classes.

You could implement something similar for your server.


As mentioned, the approach does not work for (many; all?) elementary
Python types. Thus, if the leak involves only those instances, it
cannot be detected this way.

Memory leaks are often introduced by C extensions - and do not
involve Python objects (but leak C level memory). Those, too,
cannot be analysed by Python level approaches.