[SciPy-user] mid-size outerproduct too big for memory

[Thomas Davidoff]

Thanks for the help.
The code (attached) is going to reek of poor programming skills but I 
think the dimensions on line 158 (where the program stops) are 
conformable.  I can't attach the datasets - too big.  If you are truly 
curious, the .uf3 files are at census.gov under sf3 ftp page.
As for the other processes, I don't have mucnh of anything else going 
on.  Running top on my remote mac leads to
Processes:  55 total, 2 running, 53 sleeping... 137 threads            
20:48:38
Load Avg:  0.00, 0.00, 0.00     CPU usage:  1.3% user, 4.8% sys, 93.8% idle
SharedLibs: num =  134, resident = 60.2M code, 4.59M data, 18.3M LinkEdit
MemRegions: num = 12317, resident =  124M + 10.5M private, 89.8M shared
PhysMem:   233M wired,  163M active,  207M inactive,  604M used, 1.91G free
VM: 5.08G + 95.1M   259046(0) pageins, 11945(0) pageouts

Robert Kern wrote:

> Thomas Davidoff wrote:
>
>> I tried to get the outerproduct(a,b) where a is a 23,297 by 15 matrix 
>> and b is a 15 element row vector.  This led to a memory error.  This 
>> is a pretty small dataset for empirical economics - is there a way 
>> around this? 
>
>
> 23297*15*15*8./1024/1024 = 40
>
> 40 MB. My G4-1GB memory handles this fine. I'm not sure why you are 
> seeing a memory error. Could you post your code?
>
>> Will I ever be able to do something like 1 million by 100 or is it 
>> back to Stata for me?
>
>
> 1000000*100*100*8/1024/1024/1024 == 74
>
> 74 GB. No, I don't think your machine will handle that outerproduct well.
>
>> I don't think the issue is other stuff in the program because nothing 
>> should be eating a lot of memory. 
>
>
> How about other stuff running on the system?
>
>> I noticed I get the same problem in command line python just trying a 
>> 100,000 by 100 matrix and a 1 by 100 row vector.
>> I am on a new G5 PowerMac with lots of new memory installed.
>
>

-- 
Thomas Davidoff
Assistant Professor
Haas School of Business
UC Berkeley
Berkeley, CA 94618
Phone:	(510) 643-1425
Fax:	(510) 643-7357
email:	davidoff at haas.berkeley.edu
web:	http://faculty.haas.berkeley.edu/davidoff/

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: jueround2.py
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20050424/12411904/attachment.ksh>