Why deleting large variables doesn't free up any memory?

[Oren Tirosh]

On Fri, Apr 04, 2003 at 08:13:56PM +0000, Donn Cave wrote:
> Quoth Ben Hutchings <do-not-spam-ben.hutchings at businesswebsoftware.com>:
> | In article <b6khbt$ftf$1 at panix2.panix.com>, Aahz wrote:
> |> In article <169a7283.0304040936.69e2b9a1 at posting.google.com>,
> |> sdieselil <sdieselil at yahoo.com> wrote:
> |>>
> |>> I'm using Python 2.2.2 under FreeBSD. When I create a large variable
> |>> (using for example "a = range(1000000)") 'top' command indicates that
> |>> Python ate 12MB of swap space. But when I execute "del a" nothing
> |>> happens, it doesn't release these 12MB! Using "gc.collect()" also
> |>> doesn't help. What's the problem? What happened to Python's garbage
> |>> collector?
> |> 
> |> That's got nothing to do with Python; generally speaking, once the OS
> |> allocates memory to a process, the process keeps it until it exits.
> |> However, if you allocate and de-allocate big chunks of memory, you'll
> |> notice that Python's size stays static -- the memory gets re-used.
> |
> | Actually it's an issue for the heap manager, not the OS.  Under Windows
> | and Unix at least, it is possible for a process to return memory to
> | the OS.  In practice allocation is done in chunks of at least a page
> | and a page can't be freed until every object in it has gone.  Further-
> | more the memory will probably be needed again, and if it isn't it will
> | be paged out to disk.  So most heap managers don't bother to do this,
> | except for very large allocations.
> 
> Now that's interesting.  Could you provide a reference, say a man page
> for a system call, for how a user process would return memory, on a
> UNIX platform?  (Memory that doesn't just happen to be at the top of
> the data segment, of course.)

If the memory was allocated using an anonymous mmap instead of sbrk you 
can return it to the system using munmap.

Take a look at dlmalloc. http://gee.cs.oswego.edu/dl/html/malloc.html
This malloc implementation was written by Doug Lea and used in glibc and 
several other places.  It handles large allocations above a certain 
threshold (256k by default) using an anonymous mmap.

I recommend reading the dlmalloc code to everyone. It'll make you a better 
programmer.

    Oren