[Numpy-discussion] Maximum available dimensions in numpy

[Fernando Perez]

Paul Barrett wrote:

>>>How about setting the default case to 3 or 4 dimensions and then
>>>special casing the rare higher dimensional arrays, i.e. using malloc
>>>for these situations.  The default dimension size could be a compile
>>>time option for those who routinely exceed the default size of 3 or 4.
>>
>>This seems  like premature optimization. In most cases, if you're in a
>>situation where the dimensional overhead matters (lot's of small arrays)
>>you are using Numeric/Numarray/NumPy poorly and your code is going to be
>>slow and bloated anyway. The way to get good efficiency with these
>>extensions is to do block operations on large matrices. This often
>>involves a little trickery and several extra dimensions. Reducing the
>>default matrix size down to 3 or 4 makes efficient code slower since
>>going through malloc will involve an extra dereference and probably some
>>extra branches.
> 
> 
> 
> It also avoids the possibility of running up against the maximum number of
> dimensions, while conserving memory. For those users that create a multitude
> of small arrays, the wasted memory might become important.  I only suggested
> 3 or 4 dimensions, because it would appear to cover 99% of the cases.  I
> hope those users creating the other 1%,  know what they are doing.

How about following blitz convention (since we use blitz for looping in C++ 
fast via weave)?  Blitz sets the max at 11 (I don't know if this comes from a 
Spinal Tap reference, string theory, or pure chance :).

I agree that 32 is perhaps excessive, but 4 is certainly too low a number, I 
think.  Given that most regular numerical algorithms scale exponentially with 
the number of dimensions, it's hard to think of anything useful you can do 
with a 32-dimensional array.  But there are plenty of algorithms that 
brute-force their way through problems with k^d scaling, for d=10 or so.

Cheers,

f