[Python-Dev] Problems with the Python Memory Manager

["Martin v. Löwis"]

Travis Oliphant wrote:
> As verified by removing usage of the Python PyObject_MALLOC function, it 
> was the Python memory manager that was performing poorly.   Even though 
> the array-scalar objects were deleted, the memory manager would not 
> re-use their memory for later object creation. Instead, the memory 
> manager kept allocating new arenas to cover the load (when it should 
> have been able to re-use the old memory that had been freed by the 
> deleted objects--- again, I don't know enough about the memory manager 
> to say why this happened).

One way (I think the only way) this could happen if:
- the objects being allocated are all smaller than 256 bytes
- when allocating new objects, the requested size was different
   from any other size previously deallocated.

So if you first allocate 1,000,000 objects of size 200, and then
release them, and then allocate 1,000,000 objects of size 208,
the memory is not reused.

If the objects are all of same size, or all larger than 256 bytes,
this effect does not occur.

Regards,
Martin