[SciPy-User] [SciPy-user] Problem with np.load() on Huge Sparse Matrix
Robert Elsner
mlist at re-factory.de
Fri Jun 4 08:44:08 EDT 2010
Hello,
how sparse is your matrix (NNZ)? From your code it is not clear that
mymatrix and intersection_matrix are actually the same matrices.
Cheers.
Am Donnerstag, den 03.06.2010, 23:39 -0700 schrieb Ryan R. Rosario:
> Is this a bug? Has anybody else experienced this?
>
> Not being able to load a matrix from disk is a huge limitation for me. I
> would appreciate any help anyone can provide with this.
>
> Thanks,
> Ryan
>
>
>
> Ryan R. Rosario wrote:
> >
> > Hi,
> >
> > I have a very huge sparse (395000 x 395000) CSC matrix that I cannot
> > save in one pass, so I saved the data, indices, indptr and shape in
> > separate files as suggested by Dave Wade-Farley a few years back.
> >
> > When I try to read back the indices pickle:
> >
> >>> np.save("indices.pickle", mymatrix.indices)
> >>>> indices = np.load("indices.pickle.npy")
> >>>> indices
> > array([394852, 394649, 394533, ..., 0, 0, 0], dtype=int32)
> >>>> intersection_matrix.indices
> > array([394852, 394649, 394533, ..., 1557, 1223, 285], dtype=int32)
> >
> > Why is this happening? My only workaround is to print all of entries
> > of intersection_matrix.indices to a file, and read in back which takes
> > up to 2 hours. It would be great if I could get np.load to work
> > because it is much faster.
> >
> > Thanks,
> > Ryan
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
>
More information about the SciPy-User
mailing list