[SciPy-User] memory errors when using savemat

[Tyrel Newton]

On Jul 25, 2010, at 9:49 AM, Pauli Virtanen wrote:

> Sat, 24 Jul 2010 10:37:33 -0700, Tyrel Newton wrote:
>> I'm trying to use scipy.io.savemat to export a very large set of data to
>> a .mat file. The dataset contains around 20 million floats. When I try
>> to export this to a .mat file, I get a MemoryError. The specific
>> MemoryError is:
>> 
>> File "C:\Python26\lib\site-packages\scipy\io\matlab\miobase.py", line
>> 557 in write_bytes
>> 	self.file_stream.write(arr.tostring(order='F'))
> 
> What is the complete error message? -- It typically indicates the 
> specific part in C code the error originates from. (The full traceback, 
> thanks!)

Attached is a Windows command line screenshot my colleague captured.

> 
> On the other hand, along that code path it seems the only source of a 
> MemoryError can really be a failure to allocate memory for the tostring. 
> Your data apparently needs 160 MB free for this to succeed -- which is 
> not so much. So the question comes to what is the memory usage of the 
> code when saving, compared to the available free memory?
> 

Yeah, my theory is that the tostring process is basically trying to duplicate the memory usage by creating a string that is then written to the file. This seems like an inefficient way to do it, but my understanding of the code is limited, so I'm probably missing something.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: memory_error_cropped.png
Type: image/png
Size: 108251 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20100725/d72276a5/attachment.png>