[Numpy-discussion] Numarray problem on Opteron system, Suse 9.1

[Todd Miller]

On Fri, 2004-06-11 at 09:37, Jason Ruiter wrote:
> Greetings,
> 
Hi,

I think you're getting into bleeding edge territory.   For numarray, 
the particular problem you exposed should be solved for 1.0,  but I
still think the overall 64-bit prognosis is not good.  There are API
issues in Python itself which are going to make really large arrays a
problem until they're solved; for instance, the sequence protocol
returns the length of a sequence as an int (32-bits on most platforms).

Regards,
Todd


> I'm running Suse 9.1 on a dual opteron system with 16GB of RAM.  I'm
> using Python 2.3.3, Numeric 23.1 and numarray 0.9.
> 
> I'm trying to allocate large (>4GB) arrays.  Under Numeric, I get:
> 
> Python 2.3.3 (#1, Apr  6 2004, 09:45:08)
> [GCC 3.3.3 (SuSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from Numeric import *
> >>> a=ones(2000000000,Int16);
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/usr/lib64/python2.3/site-packages/Numeric/Numeric.py", line
> 578, in ones
>     a=zeros(shape, typecode, savespace)
> MemoryError: can't allocate memory for array
> >>>
> 
> Under numarray:
> 
> Python 2.3.3 (#1, Apr  6 2004, 09:45:08)
> [GCC 3.3.3 (SuSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from numarray import *
> >>> a=ones(2000000000,Int16)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py",
> line 1111, in ones
>     retarr = _fillarray(_gen.product(shape), 1, 0, type)
>   File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py",
> line 144, in _fillarray
>     outarr = NumArray((size,), outtype)
> ValueError: new_memory: invalid region size: -294967296.
> >>>
> 
> 
> I've verified that python, Numeric, and numarray are built and linked
> against the 64bit libraries.
> 
> I've also verified that it's possible to allocate a >16GB Array with the
> following program I found on one of the debian mailling lists.
> :q:q:q!
> #include <stdio.h>
> #include <stdlib.h>
> int main()  {  size_t n;  void *p;  double gb;
>   for(gb=20;gb>.3;gb-=.5) {
>         n= 1024L * 1024L * 1024L * gb;
>         p = malloc( n );
>         printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p);
>         free(p); }  return 0;  }
> 
>  21474836480 20.0Gb (nil)
>  20937965568 19.5Gb (nil)
>  20401094656 19.0Gb (nil)
>  19864223744 18.5Gb (nil)
>  19327352832 18.0Gb (nil)
>  18790481920 17.5Gb (nil)
>  18253611008 17.0Gb (nil)
>  17716740096 16.5Gb 0x3995a02010
>  17179869184 16.0Gb 0x3995a02010
>  16642998272 15.5Gb 0x3995a02010
>  16106127360 15.0Gb 0x3995a02010
>  15569256448 14.5Gb 0x3995a02010
>  15032385536 14.0Gb 0x3995a02010
>  14495514624 13.5Gb 0x3995a02010
>  13958643712 13.0Gb 0x3995a02010
>  13421772800 12.5Gb 0x3995a02010
>  12884901888 12.0Gb 0x3995a02010
>  12348030976 11.5Gb 0x3995a02010
>  11811160064 11.0Gb 0x3995a02010
>  11274289152 10.5Gb 0x3995a02010
>  10737418240 10.0Gb 0x3995a02010
>  10200547328  9.5Gb 0x3995a02010
>   9663676416  9.0Gb 0x3995a02010
>   9126805504  8.5Gb 0x3995a02010
>   8589934592  8.0Gb 0x3995a02010
>   8053063680  7.5Gb 0x3995a02010
>   7516192768  7.0Gb 0x3995a02010
>   6979321856  6.5Gb 0x3995a02010
>   6442450944  6.0Gb 0x3995a02010
>   5905580032  5.5Gb 0x3995a02010
>   5368709120  5.0Gb 0x3995a02010
>   4831838208  4.5Gb 0x3995a02010
>   4294967296  4.0Gb 0x3995a02010
>   3758096384  3.5Gb 0x3995a02010
>   3221225472  3.0Gb 0x3995a02010
>   2684354560  2.5Gb 0x3995a02010
>   2147483648  2.0Gb 0x3995a02010
>   1610612736  1.5Gb 0x3995a02010
>   1073741824  1.0Gb 0x3995a02010
>    536870912  0.5Gb 0x3995a02010
> 
> 
> Can someone point me in the right direction?
> 
> Thanks
> Jason
-- 
Todd Miller <jmiller at stsci.edu>