when does python free memory?

[Andrae Muys]

reanes at banta-im.com (Robert Eanes) wrote in message news:<10b3f29a.0202050840.7641923a at posting.google.com>...
> Hi-
> Reading the docs and searching the list archives, it seemed that
> python should release memory as soon as the last reference to an
> object is deleted.  However, I can't seem to prove this in practice. 
> Look at this example:

This is true for 1.5.2, however later versions of python may use a
garbage collected store for some/all variables.  I don't know enough
about python's gc to discuss this.  (ps. anyone got a good reference
on a discussion of the current CPython (de-)allocator?)

<SNIP sample code demonstrating retained heap>

> So apparently python knows that the memory formerly allocated to z is
> available, but it doesn't tell the OS.  Is this the case, and if so,
> is there any way around it?  I have an application that needs to run
> 80-150 python processes that can stay around for 15min to an hour, so
> I'd like them not to hold onto the maximum amount of memory that they
> use throughout the whole life of the process.

This is a common optimisation of malloc that reduces the number of
brk/mmap calls required to service malloc/free/malloc/free/... call's.
 It can provide a signifigant performance advantage for the common
case, but can cause problems for certain unusual cases (such as yours
above).  Most platforms malloc libraries allow you to tune them on a
per-process level (normally by setting appropriate environment
variables) if required, check with your sysadmin/vendor.

To start getting a handle on this I suggest you read up on malloc/free
and their relationship with the brk(and friends), and mmap/munmap
syscall's.

  - A relatively easy read on how a performance problem successfully
diagnosed as inefficient malloc/free's, and the tuning parameters used
to solve it.  The paper discusses Tru64 and Linux, but the concepts
are generally applicable.
http://www.linuxshowcase.org/full_papers/ezolt/ezolt.pdf

  - A paper discussing various heap smashing attacks against glibc's
malloc/free algorithm.  Useful here because of it contains a
reasonably detailed discussion of Doug Lea's allocator (see sections
3.1-3.5)
http://www.phrack.org/show.php?p=57&a=8

  - Doug Lea's own paper on the design of his allocator. (I don't
personally know of any papers on the design of Solaris' allocator, but
I'm interested in knowing of one if it exists)
http://gee.cs.oswego.edu/dl/html/malloc.html

  - An online, HTMLised version of the GNU info page on tuning
parameters for glibc's malloc/free implementations.
http://docsrv.caldera.com:8457/cgi-bin/info2html?(libc)Malloc%2520Tunable%2520Parameters

Good luck

Andrae Muys