[Python-Dev] The untuned tunable parameter ARENA_SIZE

[Tim Peters]

[Larry Hastings <larry at hastings.org>]
> ...
> Yet CPython's memory consumption continues to grow.  By the time a current
> "trunk" build of CPython reaches the REPL prompt it's already allocated 16
> arenas.

I'd be surprised if that's true ;-)  The first time `new_arena()` is
called, it allocates space for a vector of 16 (INITIAL_ARENA_OBJECTS)
`arena_object` structs.  Those are tiny, and hold bookkeeping info for
the actual arenas, none of which are allocated at first.
`new_arena()` asks the system for memory for just one arena (and 15 of
its 16 initial arena_object structs remain unused, awaiting future
calls).

At the time, 16 wasn't picked because it was the minimum Python
needed, but because it was the _most_ the bulk of a variety of casual
Python scripts needed to complete.

It's certainly gotten worse on a 64-bit box.  Here on a 64-bit Win10
under Python 3.6.1 upon reaching the interactive prompt in a DOS box
(set envar PYTHONMALLOCSTATS to get this kind of output whenever
`new_arena()` is called):

# arenas allocated total           =                   14
# arenas reclaimed                 =                    5
# arenas highwater mark            =                    9
# arenas allocated current         =                    9
9 arenas * 262144 bytes/arena      =            2,359,296

# bytes in allocated blocks        =            2,166,656
# bytes in available blocks        =              140,104
0 unused pools * 4096 bytes        =                    0
# bytes lost to pool headers       =               27,648
# bytes lost to quantization       =               24,888
# bytes lost to arena alignment    =                    0
Total                              =            2,359,296

So at most 9 arenas ("highwater mark") were ever simultaneously allocated..

Change the arena size to 64MB, and most programs would stop with the
first arena allocated ;-)