[SciPy-Dev] scipy.sparse.linalg.spsolve surprising behaviour for large sparse matrices on Linux systems

[Alain Batailly]

Hello,

I am computing the solution of a linear system Ax=b with A a large
(typically 200,000 lines and columns for the associated dense matrix)
sparse matrix and b a sparse matrix of about 100 columns.

When I run my code on Windows systems (Python *2.7*, scipy *0.14.0*), the
following command

from scipy.sparse.linalg import spsolve...Temp = spsolve(A.tocsc(),b.tocsc())

runs smoothly and requires about 7 GB of ram memory.

Running the *exact same code* with the *exact same matrices* on Linux
systems (same CPU, same amount of RAM memory: 64 GB, Linux Mint *17.3*,
python *2.7*, scipy *0.13.3*) requires more than 20 GB of ram memory and it
crashes with the following error message:

<function umfpack_di_numeric at ...> failed with UMFPACK_ERROR_out_of_memory
(see 1 <http://i.stack.imgur.com/ezdPi.png>)

Because this error is os dependent, I ruled out any issue regarding the
matrices *A* and *b*, and I am trying to find a fix specific to Linux...
But I do not know where to start... Does anyone would have any idea of what
is going on ? And why would such a problem be specific to Linux systems ?

After further investigations, it seems that the latest version of BLAS on
Linux Mint is quite old: 1.8.2. On Windows, I use BLAS 1.9.1. When using
the test_numpy.py file available here:
https://gist.github.com/osdf/3842524#file-test_numpy-py I notice very
significant differences between Linux and Windows: *Linux* : version 1.8.2,
maxint 9223372036854775807, dot: 0.76 s - *Windows* : version 1.9.1, maxint
2147483647, dot: 0,037 s.

I realized that the problem may be hardware related. Indeed, an older PC,
with the exact same libraries on the same Linux Mint distribution (Rosa
17.3) provides much more satisfying results. The benchmark mentioned in the
first update gives on this old PC: *Linux* : version 1.8.2, maxint
9223372036854775807, dot: 0,054 s.

Would anybody have observed such behaviour ?

Thank you.

Alain
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20160404/8dfeca31/attachment.html>