[Baypiggies] find memory leaks in running program

[Shannon -jj Behrens]

On Tue, Dec 7, 2010 at 9:04 AM, Marco Hornung <pythonmarco at gmail.com> wrote:
> Hey,
>
> ------------------------------------------------------------------------------------------
> questions
> ------------------------------------------------------------------------------------------
> 1. What are the best tools to analyze pythons memory stack, while it is running?
> 2. Is there a possibility to analyze the memory stack of a program with external programs? (without to change the source code - I am only interested in the object size)
> 3. Can I sort of "break" into the memory to see what objects consume how much memory?
>
> ------------------------------------------------------------------------------------------
> my scenario
> ------------------------------------------------------------------------------------------
> I have to debug a multithreaded server, which is written in the twisted-framework. One of the processes has some sort of memory leak - After one of our jobs is finished the main process has still over 59% of the entire memory allocated. I will probably have to recreate our scenario and equip our server with some memory sensors - but  it takes 12h to reproduce the scenario and I will have to change the source code(at least I do not know of other options).
> Therefore I am looking for quicker possibilities to look into what causes our memory leak.

One time I found a memory leak in MySQLdb.  It was in C code, and it
only happened when you passed a certain argument to the connection
parameters that had something to do with decoding.  It would only
manifest itself after a day.  That was really, really hard to find
because the leak was invisible from Python's perspective.  My approach
was to use a process of elimination.  I would slowly remove more and
more code and try to run the code faster and faster in a loop (making
external requests to my server) in order to detect the memory leak
more quickly.  It took a few days of effort.

By the way, it might not be completely relevant, but there was a
*fantastic* talk on debugging Ruby on Rails applications using
low-level tools:

			http://scr.bi/debuggingruby
			http://bit.ly/96dVJ9

A lot of it also applies to Python, but not all of it.  That talks was
simply incredible.  The speaker had used his techniques (which
involved a huge range of tools) to find multiple memory leaks and
massive performance problems in both Ruby and Rails.

Best Regards,
-jj

-- 
In this life we cannot do great things. We can only do small things
with great love. -- Mother Teresa
http://jjinux.blogspot.com/