[SciPy-user] large arrays ( > 2 gb) in numarray?

[Francesc Altet]

Hi Lou,

If you just need to keep very large arrays and deal with them in
slices, you can use the EArray objects in PyTables:

from tables import *
from numarray import *
# creating the container
array_b = fileh.createEArray(root, 'array_b', Float64Atom(shape=(2,0,3)),
                             "Very large array",
                             Filters(complevel = 1)) # if you want compression
# adding chunks for the large array:
for i in range(100):
    a = array(..., shape=(2,1000*1000*1000,3))
    array_b.append(a)

# reading chunks of the very large array:
for i in range(100):
    a = array_b[:, 1000*1000*1000*i:1000*1000*1000*(i+1), :]
    # operations with chunk a after this...

You should be able to do this safely in PyTables as it supports 64-bit
addressing by default (although, to say the truth, I've never tried at
it, at least with EArrays). 

This is probably not what you want, but just in case.

Cheers,

A Dimecres 09 Febrer 2005 06:53, Louis Wicker va escriure:
> Hello,
>
> I am new to the list.  I  have been building a python-fortran ensemble
> Kalman filter system for assimilating radar data into convective cloud
> models.
>
> I am running into a big problem that I thought I would ask about.
>
> I am designing this system for the SGI Altix machines, which is a
> Linux64 box with > 15 Gb of memory.
>
> I ran into the problem that Numeric or Numarray will not allow me to
> allocate an array larger than 2 gb.  Is this a fundamental limitation
> of numarray/Numeric, or
> do I need to compile numarray/Numeric in some what to get it to use 64
> bit pointers.  For example, here is the failure mode...
>
>  >>> a = numarray.zeros((200,200,50,12,40), typecode='f')
>
> Traceback (most recent call last):
>    File "<stdin>", line 1, in ?
>    File "/usr/lib/python2.2/site-packages/numarray/numarraycore.py",
> line 1200, in zeros
>      retarr = NumArray(shape=shape, type=type)
> libnumarray.error: NA_updateDataPtr: error getting read buffer data ptr
>
> any ideas?  this is running python2.3 and numarray 1.1
>
> Thanks.
>
> Lou Wicker
> ------------------------------------------------------------------------
> ----
>
> |  Dr. Louis J. Wicker
> |  Research Scientist, National Severe Storms Lab
> | 1313 Halley Cir, Norman, OK 73069
> | E-mail:   Louis.Wicker at nssl.noaa.gov
> | HTTP:  www.nssl.noaa.gov/~lwicker
> | Phone:    (405) 366-0416
> | Fax:       (405) 366-0472
>
> ------------------------------------------------------------------------
> ----
>
> _______________________________________________
> SciPy-user mailing list
> SciPy-user at scipy.net
> http://www.scipy.net/mailman/listinfo/scipy-user

-- 
>qo<   Francesc Altet     http://www.carabos.com/
V  V   Cárabos Coop. V.   Enjoy Data
 ""