"Help needed - I don't understand how Python manages memory"

[Marc 'BlackJack' Rintsch]

On Sun, 20 Apr 2008 22:46:37 +1000, Hank @ITGroup wrote:

> Apology for the previous offensive title~~
> :)
> Thanks, Rintsch, Arnaud and Daniel, for replying so soon.
> 
> I redid the experiment. What following is the record -
> 
> ``starting python``                        # == Windows Task Manager: 
> Python.exe  *4,076 *K memory-usage ==
>  >>> st1='abcdefg'*999999         # == 10,952 K ==
>  >>> del st1                                 # == *4,104*K ==
>  >>> st1='abcdefg'*999999         # == 10,952 K ==
>  >>> del st1                                 # == 4,104 K ==
> 
>  >>> li = ['abcde']*999999          # == 8,024 K ==
>  >>> del li                                    # == *4,108* K ==
> 
>  >>> from nltk import FreqDist         # == 17,596 ==
>  >>> fd = FreqDist()                        # == 17,596 ==
>  >>> for i in range(999999):fd.inc(i)  # == 53,412 ==
>  >>> del fd                                       # == *28,780* ==
>  >>> fd2 = FreqDist()                       # == 28,780 ==
>  >>> for i in range(999999):fd2.inc(i)  # == 53,412 ==
>  >>> del fd2        # == 28,780 K ==
> 
>  >>> def foo():
> ...         fd3 = FreqDist()
> ...         for i in range(999999):fd3.inc(i)
> 
>  >>>  foo()         # == *28,788* K ==
> 
>  >>> def bar():
> ...         fd4 = FreqDist()
> ...         for i in range(999999):fd4.inc(i)
> ...         del fd4
>                          # == 28,788 K ==
>  >>> bar()         # == 28,788 K ==
> 
> 
> That is my question, after ``del``, sometimes the memory space returns 
> back as nothing happened, sometimes not... ...
> What exactly was happening???

Something.  Really it's a bit complex and implementation dependent.  Stop
worrying about it until it really becomes a problem.

First of all there's no guarantee that memory will be reported as free by
the OS because it is up to the C runtime library if it "gives back" freed
memory to the OS or not.  Second the memory management of Python involves
"arenas" of objects that only get freed when all objects in it are freed. 
Third some types and ranges of objects get special treatment as integers
that are allocated, some even preallocated and never freed again.  All
this is done to speed things up because allocating and deallocating loads
of small objects is an expensive operation.

Bottom line: let the Python runtime manage the memory and forget about the
``del`` keyword.  It is very seldom used in Python and if used then to
delete a reference from a container and not "bare" names.  In your `bar()`
function it is completely unnecessary for example because the name `fd4`
disappears right after that line anyway.

Ciao,
	Marc 'BlackJack' Rintsch