[Numpy-discussion] dimensions too large error

[Anne Archibald]

On 14/03/2008, Dinesh B Vadhia <dineshbvadhia at hotmail.com> wrote:

> For the following code:
>
> I = 18000
> J = 33000
> filename = 'ij.txt'
> A = scipy.asmatrix(numpy.empty((I,J), dtype=numpy.int))
>     for line in open(filename, 'r'):
>         etc.
>
> The following message appears:
>
> Traceback (most recent call last):
>     File "C:\...\....py", line 362, in <module>
>     A= scipy.asmatrix(numpy.empty((I,J), dtype=numpy.int))
>     ValueError: dimensions too large.
>
> Is there a limit to array/matrix dimension sizes?

Yes. On 32-bit machines the hardware makes it exceedingly difficult
for one process to access more than two or three gigabytes of RAM, so
numpy's strides and sizes are all 32-bit integers. As a result you
can't make arrays bigger than about 2 GB. If you need this I'm afraid
you pretty much need a 64-bit machine.

> Btw, for numpy array's, ascontiguousarray() is available to set aside
> contiguous memory.  Is there an equivalent for scipy matrix ie. an
> ascontiguousmatrix()?

That's not exactly what ascontiguousarray() is for. Normally if you
create a fresh numpy array it will be allocated as a single contiguous
block of memory, and the strides will be arranged in that special way
that numpy calls "contiguous". However, if you take a subarray of that
array - every second element, say - the resulting data is not
contiguous (in the sense that there are gaps between the elements).
Normally this is not a problem. But a few functions - mostly
handwritten C or FORTRAN code - needs contiguous arrays. The function
ascontiguousarray() will return the original array if it is contiguous
(and in C order), or make a copy if it isn't.

Numpy matrices are just a slight redefinition of the operators on an
array, so you can always convert an array to a matrix without copying.
Thus there's little cost (for large arrays) to just using
matrix(ascontiguousarray()) if you need matrices.

Good luck,
Anne