[Python-Dev] refleaks and caches

[Jeroen Ruigrok van der Werven]

-On [20080128 03:13], Christian Heimes (lists at cheimes.de) wrote:
>Do the int/float free lists cause any trouble or can they eat lots of
>memory?

I hope I am interpreting it correctly, but it seems
http://evanjones.ca/memoryallocator/ explanation on that still applies:

"The worst offenders are integers and floats. These two object types
allocate their own blocks of memory of approximately 1kB, which are
allocated with the system malloc(). These blocks are used as arrays of
integers and float objects, which avoids waste from pymalloc rounding the
object size up to the nearest multiple of 8. These objects are then linked
on to a simple free list. When an object is needed, one is taken from the
list or a new block is allocated. When an object is freed, it is returned to
the free list.

This scheme is very simple and very fast, however, it exhibits a significant
problem: the memory that is allocated to integers can never be used for
anything else. That means if you write a program which goes and allocates
1000000 integers, then frees them and allocates 1000000 floats, Python will
hold on to enough memory for 2000000 numerical objects. The solution is to
apply a similar approach as was described above. Pools could be requested
from pymalloc, so they are properly aligned. When freeing an integer or a
float, the object would be put on a free list for its specific pool. When
the pool was no longer needed, it could be returned to pymalloc.  The
challenge is that these types of objects are used frequently, so care is
required to ensure good performance.

Dictionaries and lists use a different scheme. Python always keeps a maximum
of 80 free lists and dictionaries, any extra are freed. This is not optimal
because some applications would perform better with a larger list, while
others need less. It is possible that self-tuning the list size could be
more efficient."

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
イェルーン ラウフロック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/
We have met the enemy and they are ours...