large arrays in python (scientific)

[Jason Orendorff]

> > from __future__ import generators
> > from Scientific.Functions.LeastSquares import leastSquaresFit
> > This reads the data from the file once per iteration
> > (leastSquaresFit is an iterative algorithm), which is slow;
> > but it does not store the whole data set in memory, which
> > might be helpful.
>
> Assuming that your data points are evenly distributed in a grid,
> you could
> probably trim that down so that you didn't have to explicitly store the
> (x,y,z).

Indeed, ideally:

from __future__ import generators

class MyData:
    """ Wrapper for data.

    This presents a Numeric Python array in a way that
    Scientific.Functions.LeastSquares can understand.
    """

    def __init__(self, array):
        self.array = array

    def __iter__(self):
        array = self.array
        [xr, yr, zr] = [range(N) for N in array.shape]
        for x in xr:
            for y in yr:
                for z in zr:
                    yield ((x, y, z), array[x, y, z])


> BTW- I keep reading interesting examples of generators... they seem
> interesting, and occassionally even useful.  Where can I find some good
> info on 'em?

Well... I don't know.  PEP 255; python22/lib/test/test_generators.py...
this article...

http://www-106.ibm.com/developerworks/linux/library/l-pycon.html?dwzone=linu
x

## Jason Orendorff    http://www.jorendorff.com/