[SciPy-user] mid-size outerproduct too big for memory

[Robert Kern]

Thomas Davidoff wrote:
> I tried to get the outerproduct(a,b) where a is a 23,297 by 15 matrix 
> and b is a 15 element row vector.  This led to a memory error.  This is 
> a pretty small dataset for empirical economics - is there a way around 
> this? 

23297*15*15*8./1024/1024 = 40

40 MB. My G4-1GB memory handles this fine. I'm not sure why you are 
seeing a memory error. Could you post your code?

> Will I ever be able to do something like 1 million by 100 or is 
> it back to Stata for me?

1000000*100*100*8/1024/1024/1024 == 74

74 GB. No, I don't think your machine will handle that outerproduct well.

> I don't think the issue is other stuff in the program because nothing 
> should be eating a lot of memory. 

How about other stuff running on the system?

> I noticed I get the same problem in 
> command line python just trying a 100,000 by 100 matrix and a 1 by 100 
> row vector.
> I am on a new G5 PowerMac with lots of new memory installed.

-- 
Robert Kern
rkern at ucsd.edu

"In the fields of hell where the grass grows high
  Are the graves of dreams allowed to die."
   -- Richard Harter