[SciPy-User] How to handle a scipy.io.loadmat - related bug: parts of the data inaccessible after loadmat
Propadovic Nenad
npropadovic at gmail.com
Thu Feb 23 08:12:06 EST 2017
Hello Jason, than you a lot for the answer to my post. I was aware of the
squeeze_me=True option, I think I mentioned it in my initial question post.
However, as I stated in the answer to Gregors kind answer, I actually need
a way to inspect the parts of the access path in the data structure I
import form the x.mat-file, and if I use squeeze_me=True, parts like
'RPDO2' disappear completely:
import scipy.io
y = scipy.io.loadmat("x.mat", squeeze_me=True)
cd = y['CanData']
msg = cd['msg']
print msg
Output:
((array(((array([ 61.96, 61.96, 61.96]), u'PosAct'), (array([-0.05, -0.1
, 0.3 ]), u'VelAct')),
dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0. ,
0.003968, 0.007978])),)
And I really need to be able to find it by some kind of inspection, so that
I don't return parts of the structure that don't correspond to the
intention of the person searching.
Thanks once again!
Nenad
2017-02-22 18:00 GMT+01:00 <scipy-user-request at python.org>:
> Send SciPy-User mailing list submissions to
> scipy-user at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mail.python.org/mailman/listinfo/scipy-user
> or, via email, send a message with subject or body 'help' to
> scipy-user-request at python.org
>
> You can reach the person managing the list at
> scipy-user-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of SciPy-User digest..."
>
>
> Today's Topics:
>
> 1. Re: How to handle a scipy.io.loadmat - related bug: parts of
> the data inaccessible after loadmat (Jason Sachs)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 22 Feb 2017 09:12:26 -0700
> From: Jason Sachs <jmsachs at gmail.com>
> To: SciPy Users List <scipy-user at python.org>
> Subject: Re: [SciPy-User] How to handle a scipy.io.loadmat - related
> bug: parts of the data inaccessible after loadmat
> Message-ID:
> <CAOo6sOPTMBfRATCHmoGTN8NsKcn+86dCY7fUxzB0XXcd0gZh3w at mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> ah, yes, here it is:
>
> https://docs.scipy.org/doc/scipy-0.18.1/reference/tutorial/io.html
>
> ----
>
> So, in MATLAB, the struct array must be at least 2D, and we replicate that
> when we read into Scipy. If you want all length 1 dimensions squeezed out,
> try this:
> >>>
>
> >>> mat_contents = sio.loadmat('octave_struct.mat', squeeze_me=True)>>>
> oct_struct = mat_contents['my_struct']>>> oct_struct.shape()
>
>
> On Wed, Feb 22, 2017 at 9:11 AM, Jason Sachs <jmsachs at gmail.com> wrote:
>
> > This looks familiar, I ran into this a few years ago, and if I recall
> > correctly, there is an option to loadmat to reduce array dimensions
> > appropriately. There is a "squeeze_me" option (unfortunately named...
> > should probably be deprecated in favor of "squeeze") which I think does
> > this.
> >
> > https://docs.scipy.org/doc/scipy/reference/generated/
> scipy.io.loadmat.html
> >
> > On Wed, Feb 22, 2017 at 9:02 AM, Gregor Thalhammer <
> > gregor.thalhammer at gmail.com> wrote:
> >
> >>
> >> Am 22.02.2017 um 12:02 schrieb Propadovic Nenad <npropadovic at gmail.com
> >:
> >>
> >> Hello,
> >>
> >> bear with me for the long post that follows: it took me more than a week
> >> to get this far, and I tried to compress all the relevant information
> into
> >> the post.
> >>
> >> There seems to be a bug in scipy.io.loadmat; I'll present it by a short
> >> piece of code and it's output.
> >>
> >> I create file x.mat with the following:
> >>
> >> import scipy.io
> >>
> >> d = {'CanData':
> >> {
> >> 'msg': {
> >> 'RPDO2': {
> >> 'timest': [0.0, 0.0039679999899817631,
> >> 0.0079779999941820279],
> >> 'sig': {
> >> 'VelAct': {
> >> 'Values': [-0.050000000000000003,
> >> -0.10000000000000001, 0.29999999999999999, ],
> >> 'Name': 'VelAct'
> >> },
> >> 'PosAct': {
> >> 'Values': [61.960000000000001,
> >> 61.960000000000001, 61.960000000000001, ],
> >> 'Name': 'PosAct'
> >> }
> >> }
> >> }
> >> }
> >> }
> >> }
> >> scipy.io.savemat("x.mat", d)
> >>
> >> Matlab is happy with the file and handles it the way I expect.
> >>
> >> When I read in the data stored in the file and print it out:
> >>
> >> import scipy.io
> >> y = scipy.io.loadmat("x.mat")
> >> # print y
> >> cd = y['CanData']
> >> msg = cd['msg']
> >> print msg
> >> print msg.dtype
> >> print msg.dtype.names
> >>
> >> The output is:
> >> >C:\Anaconda2\pythonw -u "test1.py"
> >> [[ array([[ ([[(array([[ ([[(array([[ 61.96, 61.96, 61.96]]),
> >> array([u'PosAct'],
> >> dtype='<U6'))]], [[(array([[-0.05, -0.1 , 0.3 ]]),
> >> array([u'VelAct'],
> >> dtype='<U6'))]])]],
> >> dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0. ,
> >> 0.003968, 0.007978]]))]],)]],
> >> dtype=[('RPDO2', 'O')])]]
> >> object
> >> None
> >>
> >> Now I've read the manual, and as I see it I have no way for me to
> access
> >> the deeper layers of data I just put in the file x.mat, although they
> are
> >> obviously right there in the data read in. Access via msg['RPDO2']
> gives:
> >> IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis
> >> (`None`) and integer or boolean arrays are valid indices.
> >>
> >>
> >> For historic reasons, in Matlab everything is at least a 2D array, even
> >> scalars. By sprinkling some [0,0] in your code you should get what you
> >> want, e.g.
> >>
> >> msg[0,0]['RPDO2'][0,0]['timest'][0,0]
> >>
> >> array([[ 0. , 0.003968, 0.007978]])
> >>
> >>
> >> Gregor
> >>
> >>
> >>
> >> If I use parameter squeeze_me=True:
> >>
> >> scipy.io.savemat("x.mat", d)
> >> y = scipy.io.loadmat("x.mat", squeeze_me=True)
> >> # print y
> >> cd = y['CanData']
> >> msg = cd['msg']
> >> print msg
> >> print msg.dtype
> >> print msg.dtype.names
> >>
> >> I get output:
> >> >C:\Anaconda2\pythonw -u "test1.py"
> >> ((array(((array([ 61.96, 61.96, 61.96]), u'PosAct'), (array([-0.05,
> >> -0.1 , 0.3 ]), u'VelAct')),
> >> dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0. ,
> >> 0.003968, 0.007978])),)
> >> object
> >> None
> >> >Exit code: 0
> >>
> >> All well, but the name 'RPDO2' disapeared from the data!
> >>
> >> Now I need this information; in future I won't handle what's put into
> >> x.mat, so I need a way to access through the data all the way down (and
> >> handle the variations that will come).
> >>
> >> I have found a workaround at:
> >> http://stackoverflow.com/questions/7008608/scipy-io-loadmat-
> >> nested-structures-i-e-dictionaries/
> >>
> >> The problem is, the workaround uses struct_as_record=False in loadmat,
> >> and which boils down to using scipy.io.matlab.mio5_params.mat_struct,
> >> and when you read the docstring of class mat_struct, it says:
> >>
> >> '''
> >> ...
> >> We deprecate this method of holding struct information, and will
> >> soon remove it, in favor of the recarray method (see loadmat
> >> docstring)
> >> '''
> >> So my questions:
> >> 1) Did I miss something? Is there a way to access the data in 'RPDO2' by
> >> using this name, without using parameter struct_as_record=False in
> loadmat?
> >> 2) If not, where do I file a bug? The workaround is five years old, so
> >> the issue seems to be in scipy for ages...
> >>
> >> (For the records, I use scipy within Anaconda2 1.4.1, under Windows, but
> >> this does not seem to matter).
> >>
> >> Thanks a lot for the answers, in advance.
> >>
> >> Nenad
> >>
> >>
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at python.org
> >> https://mail.python.org/mailman/listinfo/scipy-user
> >>
> >>
> >>
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at python.org
> >> https://mail.python.org/mailman/listinfo/scipy-user
> >>
> >>
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-user/
> attachments/20170222/e992b074/attachment-0001.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at python.org
> https://mail.python.org/mailman/listinfo/scipy-user
>
>
> ------------------------------
>
> End of SciPy-User Digest, Vol 162, Issue 6
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-user/attachments/20170223/d9f231e5/attachment.html>
More information about the SciPy-User
mailing list