Memory Leaks and Heapy

[M.-A. Lemburg]

On 2008-01-05 03:19, Yaakov Nemoy wrote:
> On Jan 4, 2008 11:56 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> The most common answer I heard was possible fragmentation, meaning
>>> there are no or few completely empty blocks to be found.  If there are
>>> no 'leaks' in the VM, then it's probably related to how memory is
>>> freed.
>> You can check for this by using a special build of Python
>> with disabled PyMalloc - Python will then use the standard
>> OS malloc for all object allocations. It still uses free lists
>> for a couple of object types, but those won't cause major
>> leak problems.
> 
> How do I go about setting this up?

There's a configure option for this:

  --with(out)-pymalloc    disable/enable specialized mallocs

>> Alternatively, try to tune the PyMalloc implementation (see
>> objmalloc.c), e.g. enable WITH_MEMORY_LIMITS.
>>
>>>> This could be caused by interned strings which are kept in a special
>>>> pool dictionary to speed up string comparisons.
>>> That's quite possible, a majority of the code is a huge number of SQL
>>> connections and code.  All string based.
>> Note that strings are normally *not* interned. However, you can
>> write code in a way that causes Python to intern more strings
>> than needed, e.g. if you dynamically compile code in your app,
>> which then causes all identifiers in the compiled code to be
>> interned.
>>
>> The interned dictionary is not exposed in Python, but you can
>> access it using a debugger via Objects/stringobject.c:interned.
> 
> That's going a bit more low level detail than I think I can handle.
> I'm not much for C programming, and i'll have a hard time sifting
> through the noise to find the signal.

Fair enough. Just wanted to give some more details as to
where to look for things that look like leaks, but are in
fact just results of internal feature of the Python
interpreter.

>>>> However, the first thing to check is whether any of the C extension
>>>> modules you are using is leaking memory. Python itself is usually
>>>> well tested for memory leaks, but this is less so for C extension
>>>> modules and it's easy to mis a few Py_DECREFs (decrementing a
>>>> Python object's reference count), causing objects to live forever.
>>> I'll try to track it down, but AFAIK, most of the code is python, and
>>> the only C code there would be is the MySQL container.  How can I
>>> debug the VM though, to determine where the leak lies?  Heapy wasn't
>>> able to tell me this, and this is the important aspect.  I'm wondering
>>> how most people go about determining the causes of leaks like these,
>>> so I can provide some accurate bug information.
>> Building Python in debug mode provides some help with this.
>> You can then detect whether objects get properly freed.
> 
> Again, what's the best way to go about doing this?

Again, configure with:

 --with-pydebug          build with Py_DEBUG defined

>> Doing explicit garbage collection via the gc module also
>> helps in narrowing down the leak.
> 
> Adding an explicit gc.collect() statement at the end of the offending
> function didn't do much to solve matters much.  It slowed the rate
> that garbage piled up, but it's not a solution.
> 
> Thanks for all the help.

You're welcome.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 07 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611