[Numpy-discussion] Re: Trying out Numeric3

[David M. Cooke]

Travis Oliphant <oliphant at ee.byu.edu> writes:

> Michiel Jan Laurens de Hoon wrote:
>
>> Travis Oliphant wrote:
>>
>>> Michiel Jan Laurens de Hoon wrote:
>>>
>>>> Another warning was that PyArrayObject's "dimensions" doesn't seem
>>>> to be an int array any more.
>>>
>>>
>>> Yes.   To allow for dimensions that are bigger than 32-bits,
>>> dimensions and strides are (intp *).  intp is a signed integer with
>>> sizeof(intp) == sizeof(void *).  On 32-bit systems, the warning
>>> will not cause problems.  We could worry about fixing it by
>>> typedefing intp to int (instead of the current long for 32-bit
>>> systems).

Why not use Py_intptr_t? It's defined by the Python C API already (in
pyport.h).

>> Do 4 gigabyte 1D numerical python arrays occur in practice? If I
>> understand correctly, the current implementation gives dimensions a
>> different pointer type on different platforms. This will break
>> extension modules on platforms other than 32-bits, as the extension
>> module expects dimensions to be a pointer to int.
>
> This is a must have.   Yes, extension modules will have to be
> recompiled and pointers changed on 64-bit platforms,  but this has to
> be done.      If you see a better solution, I'd love to hear it.  The
> earlier the better.

An array of longs would seem to be the best solution. On the two
64-bit platforms I have access to (an Athlon 64 and some Alphas),
sizeof(long) == 8, while my two 32-bit platforms (Intel x86 and
PowerPC) have sizeof(long) == 4.

For comparison, here's a list of sizes for various platforms

                  32-bit  32-bit  64-bit    64-bit
                  x86     PPC     Athlon64  Alpha
                  (Linux) (OS X)  (Linux)   (Tru64)
char              1       1       1         1
short             2       2       2         2
int               4       4       4         4
long              4       4       8         8
long long         8       8       8         8
size_t            4       4       8         8

float             4       4       4         4
double            8       8       8         8
long double       12      8       16        16

void *            4       4       8         8
function pointer  4       4       8         8

Note the three different sizes of long double (oh, fun). Also note
that size_t (which is the return type of sizeof()) is not int in
general (although lots of programs treat it like that).

Using long for the dimensions also means that converting to and from
Python ints for indices is transparent, and won't fail, as Python ints
are C longs. This is the cause of several of the 64-bit bugs I fixed
in the latest Numeric release (23.8).

[I'd help with Numeric3, but not until it compiles with fewer than
several hundred warnings -- I *really* don't want to wade through all
that.]

I've attached the program I used to generate the above numbers, if
someone wants to run it on other platforms, so we have a better idea
of what's what.

-- 
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke                      http://arbutus.physics.mcmaster.ca/dmc/
|cookedm at physics.mcmaster.ca
-------------- next part --------------
A non-text attachment was scrubbed...
Name: csizes.c
Type: text/x-csrc
Size: 522 bytes
Desc: print C type sizes
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20050322/04270373/attachment.c>