[SciPy-user] Large matrix support

[Dinesh B Vadhia]

Nathan / Alan

Here is how I'm constructing the sparse matrices:

> import numpy
> import scipy
> from scipy import sparse

> # create matrix A using the scipy.sparse coo_matrix method.  A is very sparse <10%
> ij = numpy.array(numpy.zeros((nnz, 2), dtype=numpy.int))
<snip ... read data into ij>
> row = ij[:,0]
> column = ij[:,1]
> n = ij.shape[0]
> data = scipy.ones(n, dtype=numpy.int8)

> A = sparse.coo_matrix((data, (row, column)), shape=(I,J)) # changed from dims to shape for new svn
> A = sparse.csr_matrix(X)

> f = open('A.pkl', 'wb')        # pickle the sparse matrix
> pickle.dump(A, f, 2)
> f.close()

> f = open('A.pkl', 'rb')        # unpickle the sparse matrix
> A = pickle.load(f)
> f.close()
> I = A.shape[0]
> J = A.shape[1]

<snip ... a and b are two scalar vectors>

> b = A*x

------------------------------

Message: 4
Date: Fri, 25 Apr 2008 19:16:04 -0400
From: "Alan McIntyre" <alan.mcintyre at gmail.com>
Subject: Re: [SciPy-user] Large matrix support
To: "SciPy Users List" <scipy-user at scipy.org>
Message-ID:
<1d36917a0804251616q4babd57crf8acc2b47c6b9072 at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Apr 25, 2008 at 5:01 PM, Dinesh B Vadhia
<dineshbvadhia at hotmail.com> wrote:
> My matrix sizes can be large ie. > minimum 20,000 rows x 1m columns.
>
> When just creating these data sets using Python/Numpy/Scipy, I get
> pythonw.exe errors and the program aborts.  I guess the Microsoft 32K limit
> has been reached somewhere.
>
> I haven't got any Python program to run with these data sets because of 'not
> enough memory' errors.  I'm waiting for the int8 support in the sparse
> library to hopefully resolve that issue.

Dinesh,

I apologize if this is too basic a question to ask, but are the
matrices sparse?  Could you post some sample code to show how you're
trying to construct these matrices?

Thanks,
Alan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20080425/e3e094e5/attachment.html>