Python memory deallocate

[Heiko Wundram]

Am Donnerstag 11 Mai 2006 18:07 schrieb Michele Petrazzo:
> Heiko Wundram wrote:
> > As was said before: as long as you keep a reference to an object, the
> > object's storage _will not be_ reused by Python for any other objects
> > (which is sensible, or would you like your object to be overwritten by
> > other objects before you're done with them?). Besides, even if Python did
> > free the memory that was used, the operating system wouldn't pick it up
> > (in the general case) anyway (because of fragmentation issues), so Python
> > keeping the memory in an internal free-list for new objects is a sensible
> > choice the Python developers took here.
>
> This isn't true. Just tried with python 2.5a2 and:
>
> d:\python25\python.exe
>
> >>> a = range(1000 * 100 *100) # 173 MB
> >>> del a # 122MB
>
> So now, like you saied, if I try to allocate another memory chunk,
>
> python'll re-use it... But this isn't true:
> >>> b = range(100 * 100 * 100) # 126 MB
> >>> del b # 122MB
> >>> exit() # :)
>
> d:\python25\python.exe
>
> >>> b = range(100 * 100 * 100) # 19 MB
>
> Do why python don't reuse the freed memory and re-allocate 4 MB (126 -
> 122)?

What the OS reports as the memory usage isn't actually what's allocated by 
Python objects. As Tim Peters pointed out in another post, this number just 
specifies the amount of memory the OS has given to libc, which in turn has 
given it to Python.

Basically, what you're seeing is exactly the reuse of memory that Python 
internally does. Why it allocates 4 more megabytes I wouldn't know, but as 
you can see, the range b is actually 19 megabytes in size (a little smaller, 
but >10MB), whereas it only takes up 4 megabytes when you construct it after 
you deleted the old list. That's a major difference.

I just tried the following:

modelnine at phoenix ~ $ python
Python 2.4.3 (#1, May  4 2006, 23:51:29)
[GCC 3.4.6 (Gentoo 3.4.6-r1, ssp-3.4.5-1.0, pie-8.7.9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

1000      7836  0.0  0.3  15816  3540 pts/1    S+   15:09   0:00 python

>>> x = range(1000000)

1000      7836  0.2  3.8  47656 35388 pts/1    S+   15:09   0:00 python

>>> del x

1000      7836  0.2  2.9  39840 27572 pts/1    S+   15:09   0:00 python

>>> x = range(1000000)

1000      7836  0.2  3.8  48720 35460 pts/1    S+   15:09   0:00 python

>>> del x

1000      7836  0.2  2.9  40904 27644 pts/1    S+   15:09   0:00 python

>>> x = range(1000000)

1000      7836  0.2  3.8  48720 35460 pts/1    S+   15:09   0:00 python

>>> del x

1000      7836  0.2  2.9  40904 27644 pts/1    S+   15:09   0:00 python

...

Ad infinitum (see the pattern?)

Convinced now that Python reuses memory? (see the sixth column for actual 
memory usage, the fifth is mapped memory, which contains shared libraries, 
etc.) The list is about 32MB in size (see the initial jump from 3MB to 35MB), 
but only adds about 8MB in each iteration, which are freed in each run.

--- Heiko.