Reading large multidimensional arrays off file

[José Luis]

        Hi all,
        Sometime ago, I posted a query about reading data from a
binary file, with its header and all that. Chris Barker very kindly
answered my questions, and I was a happy person. I now have a new set
of data files, and I was trying to read them in with Python. the files
are simple: there are N "lines", each with M "samples", and each
sample represents 10 signed bytes. I need to access each of the
unsigned bytes individually, and at the moment, I am doing this
through a loop, as follows (I am using the SciPy library):



from scipy import io
from scipy import plt
from scipy import *
import gui_thread

#Define the filename
fname = "/cdrom/ddata.dat"

#Get a file pointer to the file
fp = io.fopen(fname,'r',format='b')


#Skip header (30720 bytes in this case)
fp.fread(30240,'b','f')
#The image array
img=array(zeros((1280,1024),Complex)) 

for line in range(1279):
    for sample in range(1024):
        matrix=fp.fread(10,'1','f')
        #Random processing of bytes :-)
        M = StokesMatrix(matrix,scale)
        C = Stokes2Covariance(M)
        img[line,sample] = abs(C[0,0])
        
#Save the image to a matlab file to play around with later...
io.savemat('flev116-1a-c',{'img':img})


        The previous process takes ages to read the file in (of the
order of an hour for one 13 Mb file), so I was wondering why I don't
just read the whole array in, after all, I know that it is NxMx10
bytes, and I know the order. My question is fairly basic: How do I
specify the order of the data? I would like to read it in the order
shown above, into an 3-dimensional array size (10,N,M), but I am not
sure what sizes come first or second in Python.

        Many thanks in advance,
José
-- 
José L Gómez Dans			PhD student
Tel: +44 114 222 5582			Radar & Communications Group
FAX; +44 870 132 2990			Department of Electronic Engineering
					University of Sheffield UK