[Python-Dev] Arena-freeing obmalloc ready for testing

[Tim Peters]

I spent most of my PyCon sprint time so far working on Evan Jones's
arena-freeing obmalloc patch:

    http://www.python.org/sf/1123430

It's ready to test now!  The work is on a branch:

    svn+ssh://svn.python.org/python/branches/tim-obmalloc

Only obmalloc.c is changed in that branch, and you can get it directly from:

    <http://svn.python.org/view/python/branches/tim-obmalloc/Objects/obmalloc.c?rev=42760&view=log>

You should be able to drop it into a current trunk checkout without problems.

Why it's still on a branch:  obmalloc has heroically optimized "hot
paths" covering the most frequent cases, and Evan's patch kindly ;-)
avoided slowing those paths directly.  So I wasn't much worried about
speed.  However, "head review" revealed that the extremely hot-path
Py_ADDRESS_IN_RANGE() macro was no longer correct in all cases (in the
presence of free()'ing arenas), and I couldn't repair that without
adding an additional test+branch to the macro's expansion.  That means
the jury is out on speed implications, and anyone who cares about that
(including me) gets about a week to measure it (I don't want to drag
this out beyond that).

I'm optimistic, because the new test compares a quantity already being
tested by the macro, a second time against 0, and it's hard to get
cheaper than that.  However, the new branch isn't predictable, so who
knows?

Other info:  the debug "pymalloc stats" output is expanded, to track
and report the current # of arenas allocated, the highwater mark
(largest value ever seen) for the current # of arenas allocated, the
total # of times an arena was allocated, and the total # of times an
arena was free'd.

Here's output from the end of a -uall debug-build test run(*) w/ the
PYTHONMALLOCSTATS
envar set, showing that arenas do get reclaimed:

# times object malloc called       =          138,387,107
# arenas allocated total           =                1,841
# arenas reclaimed                 =                1,733
# arenas highwater mark            =                  212
# arenas allocated current         =                  108

(*) The branch was created before the sprints started, so this
   doesn't reflect the current trunk tests.

   test_subprocess was excluded, because it hangs with
   PYTHONMALLOCSTATS set (the debug output gets written
   to stderr, and at least one of the test_subprocess tests
   doesn't expect anything in the spawned process's stderr,
   the pipe it sets up for stderr fills, and both processes
   deadlock).

For simpler fun, run this silly little program, and look at memory
consumption at the prompts:

"""
x = []
for i in xrange(1000000):
   x.append([])
raw_input("full ")
del x[:]
raw_input("empty ")
"""

For example, in a release build on WinXP, VM size is about 48MB at the
"full" prompt, and drops to 3MB at the "empty" prompt.  In the trunk
(without this patch), VM size falls relatively little from what it is
at the "full" prompt (the contiguous vector holding a million
PyObject* pointers is freed, but the obmalloc arenas holding a
million+1 list objects are never freed).

For more info about the patch, see Evan's slides from _last_ year's PyCon:

    http://evanjones.ca/memory-allocator.pdf