large arrays in python (scientific)

[Dennis Roark]

>> > I have a data set that contains values for positions in 3D space.
>> > There are about 2 million data points
>> > (128x128x128).
>> >
>> > I'm trying to fit a function to the data. I want to use the
>> > LeastSquaresFit procedure in ScientificPython,
>>
>> I know nothing about Python, but you sure it's the best instrument
>> for this sort of stuff ? Sounds more like something for R to me.
>> ( http://www.cran.org ).
>> Anyway you have a huge data set .... going to need a lot of memory.
>>
>> --
>> Tae Kyon
>> Non c'è nulla di più animale
>> della coscienza pulita,
>> sul terzo pianeta del sistema solare
> 
 

For very large matrices, where most of the elements are 0, a different 
data structure than a multi-dimensional array can be much more efficient: 
look into Sparse Matrices.  They will store only those elemenents that are 
non-zero.  If you need to multiply two sparse matrices, you can take the 
transpose of one and then perform the multiplication scanning down the 
rows of each matrix.  (In a sparse matrix, it is difficult to travel down 
a particular column, but the transpose turns a column traversal into an 
easier row traversal.)  The sparse matrix saves the huge amount of memory 
that a regular matrix would use, and speeds matrix manipulation because of 
that.

-- 
Dennis Roark

denro at earthlink.net
Starting Points:
www.home.earthlink.net/~denro