Reclaiming (lots of) memory

[Nick Craig-Wood]

Thomas Rast <foo.bar at freesurf.ch.invalid> wrote:
>  My scenario is somewhat involved, so if you're in a hurry, here's an
>  abstract: I have a daemon that needs about 80MB of RAM to build its
>  internal data structures, but can pack them to 20MB for the rest of
>  its lifetime.  The other 60MB are never returned to the operating
>  system.

First a recap on how memory allocation works!  This is general - I
don't know exactly how the python memory allocation works, but there
aren't too many ways of doing it under linux.

The way malloc() and friends work under linux is to claim large blocks
of memory from the OS.  This is either done with brk()/sbrk() or with
mmmap("/dev/zero") (depending on exactly which version of glibc you
are using etc).

For brk()/sbrk() malloc keeps a single area which grows and grows.
This area *can* be shrunk, but only if there is nothing that was
allocated in the way.

mmap("/dev/zero") behaves in pretty much the same way (ie it shrinks
and grows), except for large allocation (>4k I think) the allocation
is done in its own mmap() area.  This means that when it is freed it
completely disappears.

So for either of these methods freeing memory back to the OS relies on
the fact that there has been nothing allocated that will stop the
memory shrinking back down all the way.  In a dynamic object language
like python thats pretty difficult to guarantee - perhaps impossible!

The one ray of hope is the mmap("/dev/zero") for blocks bigger than
4k.  I'm pretty sure this is how modern glibc's work by default.

What you really want to do is get the allocation of the initial dict
and all of its data to be in its own mmap() block. No idea whether
this is possible in python though!  Its the kind of thing you could do
in C++ if you could be bothered to wade through all the tedious syntax
;-)

When I've had this sort of problem in the past I've usually resorted
to an on disk database (dbm of some kind).  These are usually fast
enough.  It would have the advantage that you'd only have to parse the
log files once too.  They are reasonably fast to access too as all the
hot pages soon get into the OS page cache.

Also as suggested by another poster /usr/bin/sort is very, very good
at huge datasets using minimal memory.

>  I've thought of building and packing the structures in a fork()ed
>  process, then piping them over to the main part;

Thats a neat idea.  You could also build a dbm in the forked process.

>  is there an easier way to get my RAM back?

I don't think so, but hopefully an expert who knows something about
how python allocates its objects will speak up now!

...

Just for fun I converted your program to use anydbm.  It then took 135
seconds to run (vs 12 for your original code), but only used 4 MB of
memory top.  The values of the dbm are pickled hashes.  The keys could
be too, but I decided to just use a tab seperated string...

import random
import os
import anydbm
import cPickle

def fill(map):
    random.seed(0)
    for i in xrange(300000):
        k1 = "%s\t%s\t%s" % (random.randrange(100),
              random.randrange(100),
              random.randrange(100))
        k2 = random.randrange(100)
        if k1 in map:
            d = cPickle.loads(map[k1])
        else:
            d = dict()
        d[k2] = d.get(k2, 0) + 1
        map[k1] = cPickle.dumps(d, -1)

if __name__ == "__main__":
    show_me = 'ps v %d' % os.getpid()
    os.system(show_me)
    d = anydbm.open("ztest", 'n')
    fill(d)
    os.system(show_me)
    del d
    os.system(show_me)

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick