Trying to understand the memory occupation of big lists

[Dave Angel]

On 05/03/2013 07:24 AM, Michele Simionato wrote:
> I have a memory leak in a program using big arrays.

Actually, big lists.  Python also has arrays, and they're entirely 
different.

With the goal of debugging it I run into the memory_profiler module. 
Then I discovered something which is surprising to me. Please consider 
the following script:
>
> $ cat memtest.py
> import gc
> from memory_profiler import profile
>
>
> @profile
> def test1():
>      a = [0] * 1024 * 1024
>      del a
>      gc.collect()  # nothing change if I comment this
>
>
> @profile
> def test2():
>      for i in range(10):
>          a = [0] * 1024 * 1024
>          del a
>      gc.collect()  # nothing change if I comment this
>
>
> test1()
> test2()
>
> Here is its output, on a Linux 64 bit machine:
>
> $ python memtest.py
> Filename: memtest.py
>
> Line #    Mem usage    Increment   Line Contents
> ================================================
>       5                             @profile
>       6     9.250 MB     0.000 MB   def test1():
>       7    17.246 MB     7.996 MB       a = [0] * 1024 * 1024
>       8     9.258 MB    -7.988 MB       del a
>       9     9.258 MB     0.000 MB       gc.collect()  # nothing change if I comment this
>
>
> Filename: memtest.py
>
> Line #    Mem usage    Increment   Line Contents
> ================================================
>      12                             @profile
>      13     9.262 MB     0.000 MB   def test2():
>      14    17.270 MB     8.008 MB       for i in range(10):
>      15    17.270 MB     0.000 MB           a = [0] * 1024 * 1024
>      16    17.270 MB     0.000 MB           del a
>      17    17.270 MB     0.000 MB       gc.collect()  # nothing change if I comment this
>
> In the first case the memory is released (even if strangely not
> completely, 7.996 != 7.988), in the second case the memory is not. Why it is so? I did expect gc.collect() to free the memory but it is completely ininfluent. In the second cases there are 10 lists with 8 MB each, so
> 80 MB are allocated and 72 released, but 8 MB are still there apparently.
> It does not look like a problem of mem_profile, this is what observe with
> top too.
>
> Any ideas?
>

I haven't played with profile, so my comments are limited to the direct 
code.

gd.collect() has nothing to do in either of these functions, since the 
memory has already been released by the ref-count logic.  Only in the 
case of a circular reference is the gc.collect() useful.  If you want to 
see gc.collect() in action create two large objects that reference each 
other and a small one that references one of them.  Del the first two 
and then the third, and the memory cannot be released since the ref 
counts are nonzero.  Then do a gc.collect() which will realize that you 
have no way to reference either of the two large objects.

I suspect that profile is only looking at the memory from the point of 
view of the OS.  No block of memory can be released to the OS unless 
it's entirely freed.  My guess is that in the second case the variable i 
(or some other internal one relating to the loop) is in the same block 
with one of those lists.  The point is that CPython uses the C malloc() 
and free() functions, and they have their own limitations.  Most of the 
time when free() is called, the memory is NOT released to the OS, but is 
still made available within Python for future use.


-- 
DaveA