[SciPy-User] How to handle a scipy.io.loadmat - related bug: parts of the data inaccessible after loadmat

[Propadovic Nenad]

Hello,

bear with me for the long post that follows: it took me more than a week to
get this far, and I tried to compress all the relevant information into the
post.

There seems to be a bug in scipy.io.loadmat; I'll present it by a short
piece of code and it's output.

I create file x.mat with the following:

import scipy.io

d = {'CanData':
    {
    'msg': {
            'RPDO2': {
                'timest': [0.0, 0.0039679999899817631,
0.0079779999941820279],
                'sig': {
                    'VelAct': {
                        'Values': [-0.050000000000000003,
-0.10000000000000001, 0.29999999999999999, ],
                        'Name': 'VelAct'
                    },
                    'PosAct': {
                        'Values': [61.960000000000001, 61.960000000000001,
61.960000000000001, ],
                        'Name': 'PosAct'
                    }
                }
            }
        }
    }
}
scipy.io.savemat("x.mat", d)

Matlab is happy with the file and handles it the way I expect.

When I read in the data stored in the file and print it out:

import scipy.io
y = scipy.io.loadmat("x.mat")
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

The output is:
>C:\Anaconda2\pythonw -u "test1.py"
[[ array([[ ([[(array([[ ([[(array([[ 61.96,  61.96,  61.96]]),
array([u'PosAct'],
      dtype='<U6'))]], [[(array([[-0.05, -0.1 ,  0.3 ]]), array([u'VelAct'],
      dtype='<U6'))]])]],
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0.      ,
0.003968,  0.007978]]))]],)]],
      dtype=[('RPDO2', 'O')])]]
object
None

Now  I've read the manual, and as I see it I have no way for me to access
the deeper layers of data I just put in the file x.mat, although they are
obviously right there in the data read in. Access via msg['RPDO2'] gives:
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis
(`None`) and integer or boolean arrays are valid indices.

If I use parameter squeeze_me=True:

scipy.io.savemat("x.mat", d)
y = scipy.io.loadmat("x.mat", squeeze_me=True)
# print y
cd = y['CanData']
msg = cd['msg']
print msg
print msg.dtype
print msg.dtype.names

I get output:
>C:\Anaconda2\pythonw -u "test1.py"
((array(((array([ 61.96,  61.96,  61.96]), u'PosAct'), (array([-0.05, -0.1
,  0.3 ]), u'VelAct')),
      dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0.      ,
0.003968,  0.007978])),)
object
None
>Exit code: 0

All well, but the name 'RPDO2' disapeared from the data!

Now I need this information; in future I won't handle what's put into
x.mat, so I need a way to access through the data all the way down (and
handle the variations that will come).

I have found a workaround at:
http://stackoverflow.com/questions/7008608/scipy-io-loadmat-nested-structures-i-e-dictionaries/

The problem is, the workaround uses struct_as_record=False in loadmat, and
which boils down to using scipy.io.matlab.mio5_params.mat_struct, and when
you read the docstring of class mat_struct, it says:

'''
...
We deprecate this method of holding struct information, and will
soon remove it, in favor of the recarray method (see loadmat
docstring)
'''
So my questions:
1) Did I miss something? Is there a way to access the data in 'RPDO2' by
using this name, without using parameter struct_as_record=False in loadmat?
2) If not, where do I file a bug? The workaround is five years old, so the
issue seems to be in scipy for ages...

(For the records, I use scipy within Anaconda2 1.4.1, under Windows, but
this does not seem to matter).

Thanks a lot for the answers, in advance.

Nenad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-user/attachments/20170222/2c419a76/attachment-0001.html>