[Tutor] Problems creating very large arrays (over 10 million indices)

[DORSEY_EDMUND_K@LILLY.COM]

This is a multipart message in MIME format.

--Boundary_(ID_RCFIgHm6ndEwQkUSS7kpmA)
Content-type: text/plain; charset="us-ascii"

I tried creating a Numeric array but I couldn't get the binary data into 
it. The array module was so great cause I could just use readfile to read 
in the entire file at once.

I tried this ...

a = Numeric.ones((xsize, ysize, zsize), Numeric.Int16) #create 3d numeric 
array of type short filled with ones (placeholders)
a[0][0][0] = file.read(2) #read in two bytes and stick it in the array

Of course it says the that the array 'a' needs an int as input.  Plus if I 
do this method I need to use 3 nested for loops and it is significantly 
slower than using a normal 1d array.  Is this how I would have to do it 
with the Numeric module?  Thanks for all the input you've given. 

~Edmund





Edmund,

Given the size of your arrays and that they are 3 dimensional, you may 
want
to take a look at the Numeric package or the relatively new numarray
package.  They can handle multi-dimensional data in a very efficient 
manner.
They both can be found on sourceforge at 

http://sourceforge.net/projects/numpy/

------------------------- original message 
-----------------------------------
The data is stored as binary 16bit short to be precise.  I'm writing some 
software to increase the resolution of ct images using a subvoxel 
processing algorithm. 
I've read the original binary file sans the header into an array and then 
created a larger empty array to place the new image data that will be 
created based on the algorithm I'm working with. 
Since I'm not doing any rendering breaking the image into smaller pieces 
wouldn't help (I think) because I'd still have to process each piece of 
data.  Since it's voxel data (3d) I have to convert the 3d coordinates to 
a 1d array index.  Thus splitting the data up would be a huge headache and 

start to become conceptually very difficult.  In addition I'm jumping 
around the array (due to the fact that I'm placing 3d data in a 1d array) 
so the whole thing needs to be initialized at once.

Using the following worked and reasonably fast (about 5 seconds to 
initialize an array with close to 100 million indices)

myArray = [0] * arraySize

Thanks for the help!

~Edmund
-------------------------------------------------------------------------------
HTH,

-Reggie



--Boundary_(ID_RCFIgHm6ndEwQkUSS7kpmA)--