Memory problem

[John Machin]

Yi Xing wrote:
> Hi,
>
> I need to read a large amount of data into a list. So I am trying to
> see if I'll have any memory problem. When I do
> x=range(2700*2700*3) I got the following message:
>
> Traceback (most recent call last):
> 	File "<stdin>", line 1, in ?
> MemoryError
>
> Any way to get around this problem? I have a machine of 4G memory. The
> total number of data points (float) that I need to read is in the order
> of 200-300 millions.
>

2700*2700*3 is only 21M. Your computer shouldn't have raised a sweat,
let alone MemoryError. Ten times that got me a MemoryError on a 1GB
machine.

A raw Python float takes up 8 bytes. On a 32-bit machine a float object
will have another 8 bytes of (type, refcount). Instead of a list, you
probably need to use an array.array (which works on homogenous
contents, so it costs 8 bytes each float, not 16), or perhaps
numeric/numpy/scipy/...

HTH,
John