[Matrix-SIG] pickle dump and MemoryError

[Patrick Tufts]

Using Numeric and LinearAlgebra modules, I create a large matrix (approx 2k
x 26k) and then do a svd (singular value decomposition) on it without an
error.  svd decomposes an array into three arrays, and I can pickle and
write the first two resulting arrays, but the third causes a MemoryError.

If I don't pickle the third result from svd, the code runs without error.

According to top, the maximum memory use of the process is well under my
swap size. Use is 1.5G, machine has 1G of RAM and 3G of swap.  The disk I'm
trying to write to also has plenty of free space.

Any suggestions for how I can save this array to disk without error?  Error
traceback and code fragment at end of message.

--Pat

>Traceback (innermost last):
>  File "filter.py", line 98, in ?
>    p.dump(d0p)
>  File "/usr/local/lib/python1.5/pickle.py", line 97, in dump
>    self.save(object)
>  File "/usr/local/lib/python1.5/pickle.py", line 192, in save
>    self.save_reduce(callable, arg_tup, state)
>  File "/usr/local/lib/python1.5/pickle.py", line 218, in save_reduce
>    save(arg_tup)
>  File "/usr/local/lib/python1.5/pickle.py", line 198, in save
>    f(self, object)
>  File "/usr/local/lib/python1.5/pickle.py", line 288, in save_tuple
>    save(element)
>  File "/usr/local/lib/python1.5/pickle.py", line 198, in save
>    f(self, object)
>  File "/usr/local/lib/python1.5/pickle.py", line 270, in save_string
>    self.write(STRING + `object` + '\n')
>MemoryError

The code that generates this error is:

>import sys,string,regex
>from Numeric import *
>from LinearAlgebra import *

>[t0,s0,d0p]=singular_value_decomposition(d)

>d0pfile=open('d0ppickle','w')
>p=pickle.Pickler(d0pfile)
>p.dump(d0p)
>d0pfile.close
>
>print "pickled d0p - done"


>Python 1.5.2 (#1, Oct 11 1999, 15:01:23)  [GCC 2.8.1] on sunos5
>Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam

>uname -a
>SunOS fred 5.6 Generic_105182-05 i86pc i386 i86pc