Python resident memory retention & Evan Jones' improvements

[Andrew MacIntyre]

Matt Ernst wrote:

{...}

> I thought Evan Jones altered Python to deal with this very problem,
> and the change went into the release of 2.5.
> 
> Here is Tim Peters announcing the change:
> http://mail.python.org/pipermail/python-dev/2006-March/061991.html
> 
> He included this simple test program to show the improvement:
> """
> x = []
> for i in xrange(1000000):
>    x.append([])
> raw_input("full ")
> del x[:]
> raw_input("empty ")
> """
> 
> If you look at resident size in the "full" stage, the interpreter has
> grown to tens of megabytes. If you look at it in the "empty" stage, it
> goes back down to less than 10 megabytes. But if you run this trivial
> variation on the same program, memory use goes up and stays up:
> 
> """
> x = []
> for i in xrange(1000000):
>    x.append([])
> raw_input("full ")
> del x[:]
> for i in xrange(1000000):
>    x.append([])
> del x[:]
> raw_input("empty ")
> """
> 
> At the "empty" prompt resident memory size has not decreased. I see
> this pattern of behavior in CPython 3.1.1, 2.6.3, 2.5.2, and Jython
> 2.5.1. I have tested under 32 and 64 bit Intel Linux.
> 
> At this point I suspect that I am not going to be able to force my
> long-running processes to shrink their resident size, since I can't
> force it in much simpler tests. I am curious about why it happens
> though. That the second program should retain a larger resident memory
> footprint than the first is (to me) quite surprising.

There are two things you need to be aware of in this situation:

- not all Python's memory is allocated through Python's specialised
   malloc() - int and float objects in particular (in 2.x at least) are
   allocated directly via a privately managed free list, and any
   allocation requiring more than 256 bytes is directed to the platform
   malloc().  Any memory not allocated via Python's malloc() is not
   subject to the memory release facility referred to above.

   Python 2.6 does improve the management of memory consumed by int and
   float objects via the garbage collector.

- while Python attempts to maximally utilise memory arenas to improve
   the chances of being able to free them, Python's malloc() does not do
   any compaction of memory (ie moving allocations between arenas) within
   existing arenas.  Nor does garbage collection do this.  So fragmented
   allocations can cause the retention of nearly empty arenas.

I suspect that what you see with the second test script above is caused
by some sort of fragmentation.

My recollection of Evan's objective with his work was to deal with the
case where long running processes created lots of objects on startup,
but having initialised no longer need most of the created objects.
Without his modification, the memory would have been retained unused for
the remaining life of the process.  It also helps with cyclic bursts of
object creation/deletion.

But there are circumstances where it doesn't kick in.

To get a deeper understanding of your issue will require deeper
debugging.  I have done this at times by building Python with a wrapper
around malloc() (& friends) to log memory allocation activity.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia