Memory Leaks and Heapy

[M.-A. Lemburg]

On 2008-01-04 17:23, Yaakov Nemoy wrote:
> On Jan 4, 2008 11:10 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> If you're using lots of small objects, you may be running into a
>> problem with the Python memory allocation mechanism, pymalloc. It used
>> to not return memory to the system. In Python 2.5 (IIRC, could be
>> 2.6) this was changed to at least return completely empty blocks
>> back to the OS. For details, see Objects/obmalloc.c
> 
> The most common answer I heard was possible fragmentation, meaning
> there are no or few completely empty blocks to be found.  If there are
> no 'leaks' in the VM, then it's probably related to how memory is
> freed.

You can check for this by using a special build of Python
with disabled PyMalloc - Python will then use the standard
OS malloc for all object allocations. It still uses free lists
for a couple of object types, but those won't cause major
leak problems.

Alternatively, try to tune the PyMalloc implementation (see
objmalloc.c), e.g. enable WITH_MEMORY_LIMITS.

>> This could be caused by interned strings which are kept in a special
>> pool dictionary to speed up string comparisons.
> 
> That's quite possible, a majority of the code is a huge number of SQL
> connections and code.  All string based.

Note that strings are normally *not* interned. However, you can
write code in a way that causes Python to intern more strings
than needed, e.g. if you dynamically compile code in your app,
which then causes all identifiers in the compiled code to be
interned.

The interned dictionary is not exposed in Python, but you can
access it using a debugger via Objects/stringobject.c:interned.

>> However, the first thing to check is whether any of the C extension
>> modules you are using is leaking memory. Python itself is usually
>> well tested for memory leaks, but this is less so for C extension
>> modules and it's easy to mis a few Py_DECREFs (decrementing a
>> Python object's reference count), causing objects to live forever.
> 
> I'll try to track it down, but AFAIK, most of the code is python, and
> the only C code there would be is the MySQL container.  How can I
> debug the VM though, to determine where the leak lies?  Heapy wasn't
> able to tell me this, and this is the important aspect.  I'm wondering
> how most people go about determining the causes of leaks like these,
> so I can provide some accurate bug information.

Building Python in debug mode provides some help with this.
You can then detect whether objects get properly freed.

Doing explicit garbage collection via the gc module also
helps in narrowing down the leak.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 04 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611