Memory Leaks and Heapy

[Yaakov Nemoy]

On Jan 4, 2008 11:56 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> > The most common answer I heard was possible fragmentation, meaning
> > there are no or few completely empty blocks to be found.  If there are
> > no 'leaks' in the VM, then it's probably related to how memory is
> > freed.
>
> You can check for this by using a special build of Python
> with disabled PyMalloc - Python will then use the standard
> OS malloc for all object allocations. It still uses free lists
> for a couple of object types, but those won't cause major
> leak problems.

How do I go about setting this up?

> Alternatively, try to tune the PyMalloc implementation (see
> objmalloc.c), e.g. enable WITH_MEMORY_LIMITS.
>
> >> This could be caused by interned strings which are kept in a special
> >> pool dictionary to speed up string comparisons.
> >
> > That's quite possible, a majority of the code is a huge number of SQL
> > connections and code.  All string based.
>
> Note that strings are normally *not* interned. However, you can
> write code in a way that causes Python to intern more strings
> than needed, e.g. if you dynamically compile code in your app,
> which then causes all identifiers in the compiled code to be
> interned.
>
> The interned dictionary is not exposed in Python, but you can
> access it using a debugger via Objects/stringobject.c:interned.

That's going a bit more low level detail than I think I can handle.
I'm not much for C programming, and i'll have a hard time sifting
through the noise to find the signal.

> >> However, the first thing to check is whether any of the C extension
> >> modules you are using is leaking memory. Python itself is usually
> >> well tested for memory leaks, but this is less so for C extension
> >> modules and it's easy to mis a few Py_DECREFs (decrementing a
> >> Python object's reference count), causing objects to live forever.
> >
> > I'll try to track it down, but AFAIK, most of the code is python, and
> > the only C code there would be is the MySQL container.  How can I
> > debug the VM though, to determine where the leak lies?  Heapy wasn't
> > able to tell me this, and this is the important aspect.  I'm wondering
> > how most people go about determining the causes of leaks like these,
> > so I can provide some accurate bug information.
>
> Building Python in debug mode provides some help with this.
> You can then detect whether objects get properly freed.

Again, what's the best way to go about doing this?

> Doing explicit garbage collection via the gc module also
> helps in narrowing down the leak.

Adding an explicit gc.collect() statement at the end of the offending
function didn't do much to solve matters much.  It slowed the rate
that garbage piled up, but it's not a solution.

Thanks for all the help.