[SciPy-Dev] 64-bit sparse matrix indices

[Pauli Virtanen]

Hi,

The 32 & 64 support for sparse matrices is nearly finished, and
essentially waiting for more testing & merging:

	https://github.com/scipy/scipy/pull/442

What this will do is that sparse matrices with nnz that fit into 32 bit
use 32-bit index arrays, but those with larger nnz automatically switch
into 64-bit indices.

This means that e.g. csr_matrix.indices can be either int32 or int64. In
most cases (for sparse matrices taking less than a few gigabytes of
memory) it will be int32.

Direct matrix inversion & sparse graph algorithms are currently not
implemented for the int64 matrices. Iterative solvers and pure-Python
algorithms probably work, however.

    ***

Now would be a good moment to voice concerns on the approach, and to
actually test it in real life if you happen to have big data and enough
memory at hand (16 GB of memory is still a bit small to properly test it
in real life).

User code that relies on 32-bit indices (e.g. written in C/Cython) will
continue to work for < 2GB matrices as previously, so that this change
are mostly backward compatible.

Suggested transition route for future is to either (i) implement your
routines both in 32 and 64-bit integers, or, (ii) implement your
routines in intp size, and cast input arrays when necessary.

-- 
Pauli Virtanen