[Python-Dev] int/float freelists vs pymalloc

[M.-A. Lemburg]

On 2008-02-08 19:28, Christian Heimes wrote:
>> In addition to the pure performance aspect, there is the issue of memory
>> utilisation.  The current trunk code running the int test case in my
>> original post peaks at 151MB according to top on my FreeBSD box, dropping
>> back to about 62MB after the dict is destroyed (without a compaction).
>> The same script running on the no-freelist build of the interpreter peaks
>> at 119MB, with a minima of around 57MB.
> 
> I wonder why the free list has such a huge impact in memory usage. Int
> objects are small (4 byte pointer to type, 4 byte Py_ssize_t and 4 byte
> value). A thousand int object should consume less than 20kB including
> overhead and padding.

The free lists keep parts of the pymalloc pools alive.
Since these are only returned to the OS if the whole pool is
unused, a single object could keep 4k of memory associated
with the process.

I suppose that the remaining few MBs shown by the OS are not
really used by the process, but simply kept associated with
the process by the OS in case it quickly needs more memory.

In order to be sure about the true memory usage, you'd have
to force the OS to grab all available memory, e.g. by running
a huge process right next to the one you're testing.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 08 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611