memory problem with list creation

[Allard Warrink]

Within a python script I'm using a couple of different lists
containing a large number of floats (+8M). The execution of this
script fails because of an memory error (insufficient memory).
I thought this was strange because I delete all lists that are not
longer necessary directly and my workstation theoretically has more
than enough memory to run the script.

so I did some investigation on the memory use of the script. I found
out that when i populated the lists with floats using a for ... in
range() loop a lot of overhead memory is used and that this memory is
not freed after populating the list and is also not freed after
deleting the list.

This way the memory keeps filling up after each newly populated list
until the script crashes.


I did a couple of tests and found that populating lists with range or
xrange is responsible for the memory overhead.
Does anybody know why this happens and if there's a way to avoid this
memory problem?

First the line(s) python code I executed.
Then the memory usage of the process:
Mem usage after creation/populating of big_list
sys.getsizeof(big_list)
Mem usage after deletion of big_list

big_list = [0.0] * 2700*3250
40
35
6

big_list = [0.0 for i in xrange(2700*3250)]
40
36
6

big_list = [0.0 for i in range(2700*3250)]
145
36
110

big_list = [float(i) for i in xrange(2700*3250)]
180
36
145

big_list = [float(i) for i in range(2700*3250)]
285
36
250

big_list = [i for i in xrange(2700*3250)]
145
36
110

big_list = [i for i in range(2700*3250)]
145
36
110

big_list = []
for i in range(2700*3250):
    big_list.append(float(i))
285
36
250

big_list = []
for i in xrange(2700*3250):
    big_list.append(float(i))
180
36
145