Python memory usage

[bieffe62 at gmail.com]

On 21 Ott, 17:19, Rolf Wester <rolf.wes... at ilt.fraunhofer.de> wrote:
> Hi,
>
> I have the problem that with long running Python scripts (many loops)
> memory consumption increases until the script crashes. I used the
> following small script to understand what might happen:
>
> import gc
>
> print len(gc.get_objects())
>
> a = []
> for i in range( 4000000 ):
>     a.append( None )
> for i in range( 4000000 ):
>     a[i] = {}
>
> print len(gc.get_objects())
>
> ret = raw_input("Return:")
>
> del a
> gc.collect()
>
> print len(gc.get_objects())
>
> ret = raw_input("Return:")
>
> The output is:
> 4002706
> Return:
> 2705
> Return:
>
> When I do ps aux | grep python before the first "Return" I get:
> wester    5255 51.2 16.3 1306696 1286828 pts/4 S+   17:59   0:30 python
> memory_prob2.py
>
> and before the second one:
> wester    5255 34.6 15.9 1271784 1255580 pts/4 S+   17:59   0:31 python
> memory_prob2.py
>
> This indicates that although the garbage collector freed 4000001 objects
> memory consumption does not change accordingly.
>
> I tried the C++ code:
>
> #include <iostream>
> using namespace std;
>
> int main()
> {
>         int i;
>         cout << ":";
> //ps 1
>         cin >> i;
>
>         double * v = new double[40000000];
>         cout << ":";
> //ps 2
>         cin >> i;
>
>         for(int i=0; i < 40000000; i++)
>                 v[i] = i;
>
>         cout << v[40000000-1] << ":";
> //ps 3
>         cin >> i;
>
>         delete [] v;
>
>         cout << ":";
> //ps 4
>         cin >> i;
>
> }
>
> and got from ps:
>
> ps 1: 11184
> ps 1: 323688
> ps 1: 323688
> ps 1: 11184
>
> which means that the memery which is deallocated is no longer used by
> the C++ program.
>
> Do I miss something or is this a problem with Python? Is there any means
> to force Python to release the memory that is not used any more?
>
> I would be very appreciative for any help.
>
> With kind regards
>
> Rolf



To be sure that the deallocated memory is not cached at some level to
be reused, you could try
someting like this:

while 1:
    l = [dict() for i in range(4000000)]
    l = None # no need of gc and del

For what is worth, on my PC ( Windows XP and Python 2.5.2) the memory
usage of the process
monitored with the Task manager grows up to 600 MB before the memory
is actually released.

Note that in your example, as in mine, you do not need to call
gc.collect(), because
the huge list object is already deleted when you do "del a" ( or in my
case when I reassign "l" and the
huge list drops to 0 reference counts ). The basic memory garbage
collector in CPython
is based on reference counts; gc is only used to find and break
circular reference chains,
which your example do not create. As a proof of that, if ypu print the
retuirn value of
gc.collect (whic is the number of collected objects) you should get 0.

Ciao
------
FB