[SciPy-user] sparse csr_matrix memory error

[Nathan Bell]

On Tue, Aug 12, 2008 at 1:31 AM, Dinesh B Vadhia
<dineshbvadhia at hotmail.com> wrote:
>> nnz = 72000000
>> row = numpy.empty(nnz, dtype='intc')
>> column = numpy.empty(nnz, dtype='intc')
>> # read (i,j) data into row and column
>> data = scipy.ones(nnz, dtype='intc')
>
> For future reference, how did you arrive at the 864MB?
>

Each row and column index is 4 bytes (intc = 4 bytes).  Likewise for
the nonzero values themselves (again intc).

72e6 * (4 + 4 + 4) = 864e6 bytes

The CSR format (data,indices,indptr) is slightly smaller since the row
pointer is compressed:

indptr = 4 * (680000 + 1) # number of rows + 1
indices = 4 * 72000000  # number of nonzeros
data = 4 * 72000000  # number of nonzeros

total = 578,720,004 bytes

So combined, the two matrices require about 1.4 GB of storage.

-- 
Nathan Bell wnbell at gmail.com
http://graphics.cs.uiuc.edu/~wnbell/