Possible to set cpython heap size?

[Andrew MacIntyre]

Chris Mellon wrote:
> On 22 Feb 2007 11:28:52 -0800, Andy Watson <aldcwatson at gmail.com> wrote:
>> On Feb 22, 10:53 am, a bunch of folks wrote:
>>
>>> Memory is basically free.
>> This is true if you are simply scanning a file into memory.  However,
>> I'm storing the contents in some in-memory data structures and doing
>> some data manipulation.   This is my speculation:
>>
>> Several small objects per scanned line get allocated, and then
>> unreferenced.  If the heap is relatively small, GC has to do some work
>> in order to make space for subsequent scan results.  At some point, it
>> realises it cannot keep up and has to extend the heap.  At this point,
>> VM and physical memory is committed, since it needs to be used.  And
>> this keeps going on.  At some point, GC will take a good deal of time
>> to compact the heap, since I and loading in so much data and creating
>> a lot of smaller objects.
>>
>> If I could have a heap that is larger and does not need to be
>> dynamically extended, then the Python GC could work more efficiently.
>>
> 
> I haven't even looked at Python memory management internals since 2.3,
> and not in detail then, so I'm sure someone will correct me in the
> case that I am wrong.
> 
> However, I believe that this is almost exactly how CPython GC does not
> work. CPython is refcounted with a generational GC for cycle
> detection. There's a memory pool that is used for object allocation
> (more than one, I think, for different types of objects) and those can
> be extended but they are not, to my knowledge, compacted.
> 
> If you're creating the same small objects for each scanned lines, and
> especially if they are tuples or new-style objects with __slots__,
> then the memory use for those objects should be more or less constant.
> Your memory growth is probably related to the information you're
> saving, not to your scanned objects, and since those are long-lived
> objects I simple don't see how heap pre-allocation could be helpful
> there.

Python's internal memory management is split:
- allocations up to 256 bytes (the majority of objects) are handled by
a custom allocator, which uses 256kB arenas malloc()ed from the OS on
demand.  With 2.5 some additional work was done to allow returning
completely empty arenas to the OS; 2.3 and 2.4 don't return arenas at
all.
- all allocations over 256 bytes, including container objects that are
extended beyond 256 bytes, are made by malloc().

I can't recall off-hand whether the free-list structures for ints (and
floats?) use the Python allocator or direct malloc(); as the free-lists
don't release any entries, I suspect not.

The maximum allocation size and arena size used by the Python allocator
are hard-coded for algorithmic and performance reasons, and cannot be
practically be changed, especially at runtime.  No active compaction
takes place in arenas, even with GC.  The only time object data is
relocated between arenas is when an object is resized.

If Andy Watson is creating loads of objects that aren't being managed
by Python's allocator (by being larger than 256 bytes, or in a type 
free-list), then the platform malloc() behaviour applies.  Some platform
allocators can be tuned via environment variables and the like, in which
case review of the platform documentation is indicated.

Some platform allocators are notorious for poor behaviour in certain 
circumstances, and coalescing blocks while deallocating is one 
particularly nasty problem for code that creates and destroys lots
of small variably sized objects.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia