Python not giving free memory back to the os get's me in real problems ...

[Chris Mellon]

On 25 Apr 2007 15:04:59 GMT, TimC <leuchte at gmail.com> wrote:
> Larry Bates <larry.bates at websafe.com> wrote:
>
> > Let's see for this I need to get out my crystal ball...
> >
> > If it is a commercial application, you should contact their tech
> > support for a solution.  The problem isn't specifically a Python
> > problem but rather an implementation problem with their app.
> >
> > -Larry
>
> Well one part of the problem maybe really is on the side of this
> tool. Because the implementation of the macro functionality is
> really crappy. But unfortunately there are no competitive-product
> we could switch too, so we have to live with this poor environment.
>
> But I don't think that the WHOLE problem lies inside this tool.
> Because I read
>
> http://mail.python.org/pipermail/python-dev/2005-January/051255.html
> and
> http://evanjones.ca/python-memory.html
>
> Sadly we can't switch from Python 2.4 to 2.5 because this stupid
> macro functionality only supports 2.4.x
>
> I really know that all this is not the best setup to get happy with
> but there really is no other setup available.
>
> So I'm happy for every input all of you can give me.
>

The problem you're seeing almost certainly isn't related to the memory
arena issues referenced on the list.

"Returning memory to the OS" doesn't affect exhaustion of your virtual
address space. More likely, your nested loops are just creating more
objects than is possible to hold within a single process. That may be
because you're working with very large data sets, or because you have
some sort of ref cycle problem, or simply because you hold onto
references you no longer need. You'll need to do more work to diagnose
the exact details before you can find a fix. Contacting your vendor
support would be a really good first step.

It's possible that you have a degenerate case that triggers some of
the arena problems (like allocation 10 million integers, and then
never using them again), and thats the root of your problem. Without
knowing more about the usage pattern of  your application, theres no
way we can determine that. If you can't upgrade to python 2.5, your
only solution is going to be to rework your algorithm to not do that.
Again, contacting your vendor support is going to be the first step.