"Help needed - I don't understand how Python manages memory"

[Steve Holden]

Hank @ITGroup wrote:
> Apology for the previous offensive title~~
> :)
> Thanks, Rintsch, Arnaud and Daniel, for replying so soon.
> 
> I redid the experiment. What following is the record -
> 
> ``starting python``                        # == Windows Task Manager: 
> Python.exe  *4,076 *K memory-usage ==
>  >>> st1='abcdefg'*999999         # == 10,952 K ==
>  >>> del st1                                 # == *4,104*K ==
>  >>> st1='abcdefg'*999999         # == 10,952 K ==
>  >>> del st1                                 # == 4,104 K ==
> 
>  >>> li = ['abcde']*999999          # == 8,024 K ==
>  >>> del li                                    # == *4,108* K ==
> 
>  >>> from nltk import FreqDist         # == 17,596 ==
>  >>> fd = FreqDist()                        # == 17,596 ==
>  >>> for i in range(999999):fd.inc(i)  # == 53,412 ==
>  >>> del fd                                       # == *28,780* ==
>  >>> fd2 = FreqDist()                       # == 28,780 ==
>  >>> for i in range(999999):fd2.inc(i)  # == 53,412 ==
>  >>> del fd2        # == 28,780 K ==
> 
>  >>> def foo():
> ...         fd3 = FreqDist()
> ...         for i in range(999999):fd3.inc(i)
> 
>  >>>  foo()         # == *28,788* K ==
> 
>  >>> def bar():
> ...         fd4 = FreqDist()
> ...         for i in range(999999):fd4.inc(i)
> ...         del fd4
>                          # == 28,788 K ==
>  >>> bar()         # == 28,788 K ==
> 
> 
> That is my question, after ``del``, sometimes the memory space returns 
> back as nothing happened, sometimes not... ...
> What exactly was happening???
> 
> Best regards to all PYTHON people ~~
> !!! Python Team are great !!!
> 
It doesn't really make that much sense to watch memory usage as you have 
been doing. Your first test case appears to trigger a specific 
pathology, where the memory allocator actually returns the memory to the 
operating system when the garbage collector manages to free all of it.

Most often this doesn't happen - a chunk of memory might be 99.99% free 
but still have one small piece used, and so while there is a large 
amount of "free" memory for Python to allocate without requesting more 
process memory, this won't be reflected in any external measurement.

You are suffering from a pathological condition yourself: the desire to 
optimize performance in an area where you do not have any problems. I 
would suggest you just enjoy using Python (its memory management doesn't 
suck at all, so your title line was inflammatory and simply highlights 
your lack of knowledge) and then start to ask these questions again when 
you have a real issue that's stopping you from getting real work done.

regards
  Steve

-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/