[Numpy-discussion] dimensions too large error

[Stéfan van der Walt]

Hi Dinesh

On Fri, Mar 14, 2008 at 6:00 PM, Dinesh B Vadhia
<dineshbvadhia at hotmail.com> wrote:
> For the following code:
>
> I = 18000
> J = 33000
> filename = 'ij.txt'
> A = scipy.asmatrix(numpy.empty((I,J), dtype=numpy.int))
>     for line in open(filename, 'r'):
>         etc.
>
> The following message appears:
>
> Traceback (most recent call last):
>     File "C:\...\....py", line 362, in <module>
>     A= scipy.asmatrix(numpy.empty((I,J), dtype=numpy.int))
>     ValueError: dimensions too large.
>
> Is there a limit to array/matrix dimension sizes?

You are trying to allocate a contiguous block of memory of roughly
2.2Gb.  I'm wondering whether you have enough memory available, and
whether that memory is not already fragmented?

If your matrix is not dense, you can use the sparse matrix structures
from scipy.sparse to represent all the non-zeros.

Regards
Stéfan