[SciPy-User] Multiplying very large matrices

[eat]

kunal ghosh <kunal.t2 <at> gmail.com> writes:

Hi,
> 
> 
> Hi all,
> while implementing Locality Preserving Projections ,
> at one point i have to perform X L X.transpose()
> these matrices are large (32256 x 32256) so i get "out of memory" error.
> 
> 
> I assume, as the dataset gets larger one would come across this problem , 
how would
> one go about solving this ? Is there a common trick that is used to deal 
with such problems ?
> Or the workstation calculating these problems needs to have HUGE  amounts of 
physical memory ?
> 
> 
> I am using python and numpy / scipy-- regards-------Kunal GhoshDept of 
Computer Sc. & Engineering.Sir MVITBangalore,Indiapermalink: 
member.acm.org/~kunal.t2Blog:kunalghosh.wordpress.comWebsite:www.kunalghosh.net
46.net
Perhaps some linear algebra will help you to rearange the calculations, 
especially if your matrices are not full rank.

Forexample projection to subspace (M_hat= PM):
In [1]: M= randn(1e5, 1e1)

In [2]: U, s, V= svd(M, full_matrices= False)

In [3]: U.shape
Out[3]: (100000, 10)

In [4]: timeit dot(U, dot(U.T, M))
10 loops, best of 3: 45.2 ms per loop

In [5]: timeit # dot(dot(U, U.T), M))
# would consume all memory and even if had enough memory it would be very slow


My 2 cents,
eat
> 
> 
> 
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>