large arrays in python (scientific)

[Bill Tate]

markrgrant at yahoo.com (Mark Grant) wrote in message news:<e63502ca.0201071525.19d629a6 at posting.google.com>...
> I'm new to python programming.  I was hoping I could get some feedback
> about some difficulty I'm having
> creating a large array in python. 
> 
> I have a data set that contains values for positions in 3D space. 
> There are about 2 million data points
> (128x128x128).
> 
> I'm trying to fit a function to the data. I want to use the
> LeastSquaresFit procedure in ScientificPython,
> which takes an array of elements of the format:
> 
> [[(xposition1, yposition1, zposition1), value1],
> [[(xposition2, yposition2, zposition2), value2],
> ...,
> ...,
> ]
> 
> When I try to create this array, I create about a million of the
> elements, and then the script slows down and
> eventually stops.  I'm not sure why this is happening. 
> 
> Thanks for any help.
> 
> Mark Grant
> markrgrant at yahoo.com

Mark,
Are you sure you want to use a least squares fit to begin with???  The
~ 2 million data points defines a bounding volume and any prediction
of points within that volume based on a least squares fit is likely to
have very large error terms - your RMSE is likely to be very high and
I suspect your R2 value for least squares fit is probably going to be
very low.

I don't know what the nature of your problem is, e.g., whether you are
working with an irregular surface (like say topography data),
scattered 3-D points, or points that define something akin to a
contiguous or piece-wise contiguous surface, so its difficult to
suggest a practical alternative.  In any event, I think working with
the full 2 million data points at one time is probably not practical. 
If you can provide more details about the nature of the data you are
working with, I imagine you'll get more feedback in terms of useful
alternatives.  Depending on the kind of data you are working with,
there may be solution available that doesn't constitute much more work
that what is needed to perform a LSF.

Bill