Python memory handling

[Andrew MacIntyre]

frederic.pica at gmail.com wrote:

> Using the same file with cElementTree took me 217 Mb, with no
> unreachable object.
> For me it's not a good behavior, it's not a good way to let the system
> swap this unused memory instead of freeing it.
> I think it's a really good idea to have a memory pool for performance
> reason, but why is there no 'free block' limit ?
> Python is a really really good language that can do many things in a
> clear, easier and performance way I think. It has always feet all my
> needs. But I can't imagine there is no good solution for that problem,
> by limiting the free block pool size or best, letting the user specify
> this limit and even better, letting the user completely freeing it
> (with also the limit manual specification)
> 
> Like:
> import pool
> pool.free()
> pool.limit(size in megabytes)
> 
> Why not letting the user choosing that, why not giving the user more
> flexibility ?

Because its not easy, and its an unusual edge case that hasn't attracted
developer effort (the PyMalloc change for 2.5 was contributed by someone
who desperately needed it, not a core Python developer; it was also a
non-trivial effort to get right).

You should also appreciate something about PyMalloc: it only handles 
allocation requests of 256 bytes or smaller, and this limitation is part
of PyMalloc's design.

If most of your allocations are >256 bytes, you're at the mercy of the 
platform malloc and heap fragmentation can be a killer.  This is probably
why the getlines() approach mentioned would appear to relinquish (most
of) the memory: the list was probably comprised mostly of PyMalloc
allocations.

I haven't checked, but cElementTree may internally not be using PyMalloc
anyway, as the package is stated to be usable back to Python 1.5 - long
before the current allocation management came into effect.  In which
case, you're at the mercy of the platform malloc...  The pure Python 
ElementTree might play more your way, at a performance cost.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia