python gc performance in large apps

[Neal Norwitz]

Jp Calderone wrote:
> On Fri, 21 Oct 2005 16:13:09 -0400, Robby Dermody <robbyd at u20.org> wrote:
> >
> > [snip - it leaks memory]
>
> One thing to consider is that the process may be growing in size, not because garbage objects are not being freed, but because objects which should be garbage are being held onto by application-level code.

This is a big problem with Java too.  It's also likely to be a large
source of the memory growth here given that there isn't much cylcic
garbage.  I'm assuming that memory leaks in the python core are going
to be a small percentage of the total.  (Probably also true even if
there are memory leaks in Twisted, etc)  It's so easy to keep data
around you don't realize.

I don't have any particular insight into this problem.  I think Zope
servers can run a long without similar issues, so I think (certainly
hope) it's not endemic to the Python core.  I don't recall any
significant memory leaks fixed between 2.3 and current CVS.  But it
would be interesting to try your app on 2.4 at least to see if it
works.  CVS would also be interesting.

You might want to consider building your own python configuring
--with-pydebug.  This will cause your program to run slower and consume
more memory, but it has additional information available to help find
reference leaks.

Definitely also run under valgrind if possible.  Given the size, I
don't know if electric fence or dbmalloc are realistic options.

Feel free to mail me if you need help with valgrind etc.  I'm very
curious what the root cause of your problem is.  It's possible you are
exercising code in python that isn't commonly used and so we haven't
found a problem yet.  Also consider looking into the issues of the
third party libraries.  Jp mentioned some problems with Twisted stuff.

It would be good if you could provide small test cases that create the
problems you have encountered.

n