python gc performance in large apps

[Jp Calderone]

On Fri, 21 Oct 2005 16:13:09 -0400, Robby Dermody <robbyd at u20.org> wrote:
>
>Hey guys (thus begins a book of a post :),
>
>I'm in the process of writing a commercial VoIP call monitoring and
>recording application suite in python and pyrex. Basically, this
>software sits in a VoIP callcenter-type environment (complete with agent
>phones and VoIP servers), sniffs voice data off of the network, and
>allows users to listen into calls. It can record calls as well. The
>project is about a year and 3 months in the making and lately the
>codebase has stabilized enough to where it can be used by some of our
>clients. The entire project has about 37,000 lines of python and pyrex
>code (along with 1-2K lines of unrelated java code).
>
> [snip - it leaks memory]

One thing to consider is that the process may be growing in size, not because garbage objects are not being freed, but because objects which should be garbage are being held onto by application-level code.

gc.objects may be useful for determining if this is the case, and gc.get_objects() may be useful for discovering what kinds of objects are piling up.  These may give you a hint as to where to look to allow these objects to be released, if this is the problem.

Of course, it's also possible one of the libraries you are using is either leaking objects in this fashion, or for the extension modules, may just be leaking memory.  The above techniques may help you find an object leak, but they won't help you find a memory leak.  For this, you might give Valgrind a try (use the suppression file in Python CVS to get rid of the spew PyMalloc and friends generate).

Also, I can point out two things: Twisted's URL parsing extension leaked some memory in 1.3, but has been fixed since 2.0; and Nevow 0.4.1 made it easy to write applications that leaked several page objects per request, which has been fixed since 0.5.  If you're using either of these older versions, upgrading may fix your difficulties.

Hope this helps,

Jp