Tracking down memory leaks?

[Steven D'Aprano]

On Sun, 12 Feb 2006 05:11:02 -0800, MKoool wrote:

> I have an application with one function called "compute", which given a
> filename, goes through that file and performs various statistical
> analyses.  It uses arrays extensively and loops alot.  it prints the
> results of it's statistical significance tests to standard out.  Since
> the compute function returns and I think no variables of global scope
> are being used, I would think that when it does, all memory returns
> back to the operating system.

I may be mistaken, and if so I will welcome the correction, but Python
does not return memory to the operating system until it terminates.

Objects return memory to Python when they are garbage collected, but not
the OS.


> Instead, what I see is that every iteration uses several megs more.
> For example, python uses 52 megs when starting out, it goes through
> several iterations and I'm suddenly using more than 500 megs of ram.
> 
> Does anyone have any pointers on how to figure out what I'm doing
> wrong?

How big is the file you are reading in? If it is (say) 400 MB, then it is
hardly surprising that you will be using 500MB of RAM. If the file is 25K,
that's another story.

How are you storing your data while you are processing it? I'd be looking
for hidden duplicates.

I suggest you re-factor your program. Instead of one giant function, break
it up into lots of smaller ones, and call them from compute. Yes, this
will use a little more memory, which might sound counter-productive at the
moment when you are trying to use less memory, but in the long term it
will allow your computer to use memory more efficiently (it is easier to
page small functions as they are needed than one giant function), and it
will be much easier for you to write and debug when you can isolate
individual pieces of the task in individual functions.

Re-factoring will have another advantage: you might just find the problem
on your own.


-- 
Steven.