Swap memory in Python ? - three questions

[Scott David Daniels]

Robert LaMarca wrote:
> Hi, 
> 
> I am using numpy and wish to create very large arrays.   My system is AMD 64 x 2 Ubuntu 8.04.  Ubuntu should be 64 bit. I have 3gb RAM and a 15 GB swap drive.  
> 
> The command I have been trying to use is; 
> g=numpy.ones([1000,1000,1000],numpy.int32)
> 
> This returns a memory error.  
> A smaller array ([500,500,500]) worked fine.. 
> Two smaller arrays again crashed the system.
> 
> So... I did the math.  a 1000x1000x1000 array at 32 bits should be around 4gb RAM... Obviously larger than RAM, but much smaller than the swap drive.
> 
> 1. So... does Numpy have a really lot of overhead? Or is my system just not somehow getting to make use of the 15gb swap area. 
> 2. Is there a way I can access the swap area, or direct numpy to do so? Or do I have to write out my own numpy cache system... 
> 3. How difficult is it to use data compression internally on numpy arrays? 
> 
> thanks very much
> Robert
There are architectural issues that you should not expect (and really
do not want) the libraries / languages / os to handle for you
automatically.  The reason you don't even _want_ an automatic solution
is that the computer has no understanding of the problem you are trying
to solve.  If the machine guesses wrong, it will be doing computations
at I/O speeds, and for 4G of data, that will result in geological
computing times.

On big problems the programming issue is how to break the problem into
tractable sub-problems (of feasible computing size), and how to stitch
those results together.  This is a place to apply human intellect, not
machine effort.  So, sorry, there is no way to solve the problem without
understanding the field it occurs in and the question being addressed
by the code.

--Scott David Daniels
Scott.Daniels at Acm.Org