Memory problem

[bearophileHUGS at lycos.com]

Yi Xing wrote:
> I need to read a large amount of data into a list. So I am trying to
> see if I'll have any memory problem. When I do
> x=range(2700*2700*3) I got the following message:
> Traceback (most recent call last):
> 	File "<stdin>", line 1, in ?
> MemoryError
> Any way to get around this problem? I have a machine of 4G memory. The
> total number of data points (float) that I need to read is in the order
> of 200-300 millions.

If you know that you need floats only, then you can use a typed array
(an array.array) instead of an untyped array (a Python list):

import array
a = array.array("f")

You can also try with a numerical library like scipy, it may support up
to 2 GB long arrays.

Bye,
bearophile