Memory problem

[Yi Xing]

I used the array module and loaded all the data into an array. 
Everything works fine now.
On Aug 14, 2006, at 4:01 PM, John Machin wrote:

> Yi Xing wrote:
>> Thanks! I just found that that I have no problem with
>> x=[[10.0]*2560*2560]*500, but x=range(1*2560*2560*30) doesn't work.
>>
>
> range(1*2560*2560*30) is creating a list of 196M *unique* ints.
> Assuming 32-bit ints and pointers: that's 4 bytes each for the value, 4
> for the type pointer, 4 for the refcount and  4 for the actual list
> element (a pointer to the 12-byte object). so that's one chunk of
> 4x196M = 786MB of contiguous list, plus 196M chunks each whatever size
> gets allocated for a request of 12 bytes. Let's guess at 16. So the
> total memory you need is 3920M.
>
> Now let's look at [[10.0]*2560*2560]*500.
> Firstly that creates a tiny list [10.0]. then you create a list that
> contains 2560*2560 = 6.5 M references to that *one* object containing
> 10.0. That's 26MB. Then you make a list of 500 references to that big
> list. This new list costs you 2000 bytes. Total required: about 26.2MB.
> The minute you start having non-unique numbers instead of 10.0, this
> all falls apart.
>
> In any case, your above comparison is nothing at all to do with the
> solution that you need, which as already explained will involve
> array.array or numpy.
>
> What you now need to do is answer the questions about your pagefile
> etc.
>
> Cheers,
> John
>
> -- 
> http://mail.python.org/mailman/listinfo/python-list
>