[SciPy-User] scipy.io.loadmat throws TypeError with large files
Richard Llewellyn
llewelr at gmail.com
Wed Aug 7 15:15:31 EDT 2013
Hi,
I get this or similar (different integer than 75724 in error) exceptions
when loading a sparse matrix (CSC) saved with savemat, all default options.
>>> m = loadmat('my_large_mat.mat')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio.py",
line 176, in loadmat
matfile_dict = MR.get_variables(variable_names)
File
"/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio5.py",
line 274, in get_variables
hdr, next_position = self.read_var_header()
File
"/home/richard/venv3.3/lib/python3.3/site-packages/scipy/io/matlab/mio5.py",
line 236, in read_var_header
raise TypeError('Expecting miMATRIX type here, got %d' % mdtype)
TypeError: Expecting miMATRIX type here, got 75724
here the matrix was:
> matrix
<400000x4176 sparse matrix of type '<class 'numpy.uint8'>'
with 934099575 stored elements in Compressed Sparse Column format>
and looks fine before saving.
It looks as if this only occurs when the saved matrix file size is > 4GB --
at least I haven't seen it with files in the 3GB range.
64 bit Linux.
Not a crisis, as I am chunking anyway, so I can just chunk smaller, but
when I get more RAM would be nice to bump it up to 8 GB files or so.
Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20130807/c4684697/attachment.html>
More information about the SciPy-User
mailing list