Python memory handling

[frederic.pica at gmail.com]

On 31 mai, 17:29, "Josh Bloom" <joshbl... at gmail.com> wrote:
> If the memory usage is that important to you, you could break this out
> into 2 programs, one that starts the jobs when needed, the other that
> does the processing and then quits.
> As long as the python startup time isn't an issue for you.
>
> On 31 May 2007 04:40:04 -0700, frederic.p... at gmail.com
>
> <frederic.p... at gmail.com> wrote:
> > Greets,
>
> > I've some troubles getting my memory freed by python, how can I force
> > it to release the memory ?
> > I've tried del and gc.collect() with no success.
> > Here is a code sample, parsing an XML file under linux python 2.4
> > (same problem with windows 2.5, tried with the first example) :
> > #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb shared
> > #Usinghttp://www.pixelbeat.org/scripts/ps_mem.pyto get memory
> > information
> > import cElementTree as ElementTree #meminfo: 2.3 Mb private, 1.6 Mb
> > shared
> > import gc #no memory change
>
> > et=ElementTree.parse('primary.xml') #meminfo: 34.6 Mb private, 1.6 Mb
> > shared
> > del et #no memory change
> > gc.collect() #no memory change
>
> > So how can I free the 32.3 Mb taken by ElementTree ??
>
> > The same problem here with a simple file.readlines()
> > #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb shared
> > import gc #no memory change
> > f=open('primary.xml') #no memory change
> > data=f.readlines() #meminfo: 12 Mb private, 1.4 Mb shared
> > del data #meminfo: 11.5 Mb private, 1.4 Mb shared
> > gc.collect() # no memory change
>
> > But works great with file.read() :
> > #Python interpreter memory usage : 1.1 Mb private, 1.4 Mb shared
> > import gc #no memory change
> > f=open('primary.xml') #no memory change
> > data=f.read() #meminfo: 7.3Mb private, 1.4 Mb shared
> > del data #meminfo: 1.1 Mb private, 1.4 Mb shared
> > gc.collect() # no memory change
>
> > So as I can see, python maintain a memory pool for lists.
> > In my first example, if I reparse the xml file, the memory doesn't
> > grow very much (0.1 Mb precisely)
> > So I think I'm right with the memory pool.
>
> > But is there a way to force python to release this memory ?!
>
> > Regards,
> > FP
>
> > --
> >http://mail.python.org/mailman/listinfo/python-list


Yes it's a solution, but I think it's not a good way, I did'nt want to
use bad hacks to bypass a python specific problem.
And the problem is everywhere, every python having to manage big
files.
I've tried xml.dom.minidom using a 66 Mb xml file => 675 Mb of memory
that will never be freed. But that time I've got many unreachable
object when running gc.collect()
Using the same file with cElementTree took me 217 Mb, with no
unreachable object.
For me it's not a good behavior, it's not a good way to let the system
swap this unused memory instead of freeing it.
I think it's a really good idea to have a memory pool for performance
reason, but why is there no 'free block' limit ?
Python is a really really good language that can do many things in a
clear, easier and performance way I think. It has always feet all my
needs. But I can't imagine there is no good solution for that problem,
by limiting the free block pool size or best, letting the user specify
this limit and even better, letting the user completely freeing it
(with also the limit manual specification)

Like:
import pool
pool.free()
pool.limit(size in megabytes)

Why not letting the user choosing that, why not giving the user more
flexibility ?
I will try later under linux with the latest stable python

Regards,
FP