Memory Leak

[Peter Hansen]

Guy wrote:
> 
> It might take me a little time to explain this but here goes.

Thanks for the details you did provide; it's more than most manage!

> Firstly I'm not using the latest upto date python releases so my first
> plan is to try more upto date rels of python and win32 libs, as this
> has fixed problems for me in the past (Don't like doing this as it
> usally breaks stuff and sys admin are slow to update software,
> although they do a cracking job, just covering my back.)

Definitely do that.  There have been memory leaks in past versions
(and possibly are in the latest too, but it's perhaps less likely unless
you are using some of the newest features, which you obviously aren't).

I'd even suggest that doing much before you try that would be a waste.

> The problems occure when I build more than one flavour of the product,
> (there are 16 possible flavours and about 50 modules which get turned
> into libs and exe's  this means theres 800 processes to be set off one
> after the other or if using a clean build 1600 process and thats just
> for win32, to build one flavour it takes about 40 mins on my comp.)
> 
> When using the build script it keeps logs of what its built and
> records all output from the process all these python objs and lists
> are destroyed by using del to try and free up memory, this never seems
> to have that much effect the memory just keeps getting used up until
> usally the script or python crashes due to the lack of mem.

Memory leaks are extremely difficult to debug, perhaps more difficult 
than any other problem except race conditions.

Thankfully, real memory leaks are very rare.  I don't think you've
described enough to prove that you actually have a memory leak.  What
you are describing sounds more like you're just using up all the 
resources too quickly, which is more of a capacity (or design) problem
than anything else.

More importantly, you've got way too many variables involved to be
able to troubleshoot this, if it's really a memory leak.  You absolutely
will have to simplify the setup, while keeping the problem occurring,
in order to solve this issue.  If this is a memory leak involving Python,
you will be able to reproduce it with a program that involves nothing
but Python, and perhaps a small external dummy program that you launch
repeatedly in the other processes.

You shouldn't have to bother with "del" at all, since all that does is
release the name from the object it's bound to (or vice versa) and if
there are no other names bound to the object, Python will free the memory
anyway.  (Note that it might not return the memory to the OS, however.
You can't force that to happen, either.)  Doing a "del" followed by 
reassigning the name to a new object is basically a waste of time.

Consider this.  From the looks of things, you are doing so much that
is actually *outside* of the scope of Python that you might simply
be running out of memory elsewhere.

Also consider whether this might be due to memory fragmentation rather
than true leaking.  You might need to analyze your pattern of memory 
usage to see whether this could be the case.

-Peter