Memory Leaks and Heapy

[Yaakov Nemoy]

On Jan 4, 2008 11:10 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> If you're using lots of small objects, you may be running into a
> problem with the Python memory allocation mechanism, pymalloc. It used
> to not return memory to the system. In Python 2.5 (IIRC, could be
> 2.6) this was changed to at least return completely empty blocks
> back to the OS. For details, see Objects/obmalloc.c

The most common answer I heard was possible fragmentation, meaning
there are no or few completely empty blocks to be found.  If there are
no 'leaks' in the VM, then it's probably related to how memory is
freed.

> This could be caused by interned strings which are kept in a special
> pool dictionary to speed up string comparisons.

That's quite possible, a majority of the code is a huge number of SQL
connections and code.  All string based.

> However, the first thing to check is whether any of the C extension
> modules you are using is leaking memory. Python itself is usually
> well tested for memory leaks, but this is less so for C extension
> modules and it's easy to mis a few Py_DECREFs (decrementing a
> Python object's reference count), causing objects to live forever.

I'll try to track it down, but AFAIK, most of the code is python, and
the only C code there would be is the MySQL container.  How can I
debug the VM though, to determine where the leak lies?  Heapy wasn't
able to tell me this, and this is the important aspect.  I'm wondering
how most people go about determining the causes of leaks like these,
so I can provide some accurate bug information.

-Yaakov