[SciPy-User] SciPy-User Digest, Vol 162, Issue 5
Propadovic Nenad
npropadovic at gmail.com
Thu Feb 23 07:50:16 EST 2017
Hello Gregor,
thank you for the answer.
I was aware of the need to access "Matlab scalars" in scipy by "sprinkling
in [0,0]" in the access path. This, however, does not solve my problem, as
I might get parts of the access path with some kind of wildcard. For
example, I might get the 'RPDO2'-part in the form of '*DO2'. As I have no
way to inspect the names of the sub-structures from the imported Matlab
data, I can't even try to build all the possible names with
'*DO2', as their number is infinite.
So, as I said, if I don't know exactly, that 'RPDO2' will be part of my
path, this does not save me.
(I can assure that the requirement to get parts of the access path in such
a vague form makes sense in the context of the environment of the client I
work for).
Regards,
Nenad
2017-02-22 17:11 GMT+01:00 <scipy-user-request at python.org>:
> Send SciPy-User mailing list submissions to
> scipy-user at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mail.python.org/mailman/listinfo/scipy-user
> or, via email, send a message with subject or body 'help' to
> scipy-user-request at python.org
>
> You can reach the person managing the list at
> scipy-user-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of SciPy-User digest..."
>
>
> Today's Topics:
>
> 1. Re: How to handle a scipy.io.loadmat - related bug: parts of
> the data inaccessible after loadmat (Gregor Thalhammer)
> 2. Re: How to handle a scipy.io.loadmat - related bug: parts of
> the data inaccessible after loadmat (Jason Sachs)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 22 Feb 2017 17:02:53 +0100
> From: Gregor Thalhammer <gregor.thalhammer at gmail.com>
> To: SciPy Users List <scipy-user at python.org>
> Subject: Re: [SciPy-User] How to handle a scipy.io.loadmat - related
> bug: parts of the data inaccessible after loadmat
> Message-ID: <905AD1C7-BA68-47AE-B728-F88CC4F89CAD at gmail.com>
> Content-Type: text/plain; charset="us-ascii"
>
>
> > Am 22.02.2017 um 12:02 schrieb Propadovic Nenad <npropadovic at gmail.com>:
> >
> > Hello,
> >
> > bear with me for the long post that follows: it took me more than a week
> to get this far, and I tried to compress all the relevant information into
> the post.
> >
> > There seems to be a bug in scipy.io.loadmat; I'll present it by a short
> piece of code and it's output.
> >
> > I create file x.mat with the following:
> >
> > import scipy.io <http://scipy.io/>
> >
> > d = {'CanData':
> > {
> > 'msg': {
> > 'RPDO2': {
> > 'timest': [0.0, 0.0039679999899817631,
> 0.0079779999941820279],
> > 'sig': {
> > 'VelAct': {
> > 'Values': [-0.050000000000000003,
> -0.10000000000000001, 0.29999999999999999, ],
> > 'Name': 'VelAct'
> > },
> > 'PosAct': {
> > 'Values': [61.960000000000001,
> 61.960000000000001, 61.960000000000001, ],
> > 'Name': 'PosAct'
> > }
> > }
> > }
> > }
> > }
> > }
> > scipy.io.savemat("x.mat", d)
> >
> > Matlab is happy with the file and handles it the way I expect.
> >
> > When I read in the data stored in the file and print it out:
> >
> > import scipy.io <http://scipy.io/>
> > y = scipy.io.loadmat("x.mat")
> > # print y
> > cd = y['CanData']
> > msg = cd['msg']
> > print msg
> > print msg.dtype
> > print msg.dtype.names
> >
> > The output is:
> > >C:\Anaconda2\pythonw -u "test1.py"
> > [[ array([[ ([[(array([[ ([[(array([[ 61.96, 61.96, 61.96]]),
> array([u'PosAct'],
> > dtype='<U6'))]], [[(array([[-0.05, -0.1 , 0.3 ]]),
> array([u'VelAct'],
> > dtype='<U6'))]])]],
> > dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0. ,
> 0.003968, 0.007978]]))]],)]],
> > dtype=[('RPDO2', 'O')])]]
> > object
> > None
> >
> > Now I've read the manual, and as I see it I have no way for me to
> access the deeper layers of data I just put in the file x.mat, although
> they are obviously right there in the data read in. Access via msg['RPDO2']
> gives:
> > IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis
> (`None`) and integer or boolean arrays are valid indices.
>
> For historic reasons, in Matlab everything is at least a 2D array, even
> scalars. By sprinkling some [0,0] in your code you should get what you
> want, e.g.
>
> msg[0,0]['RPDO2'][0,0]['timest'][0,0]
> array([[ 0. , 0.003968, 0.007978]])
>
> Gregor
>
>
> >
> > If I use parameter squeeze_me=True:
> >
> > scipy.io.savemat("x.mat", d)
> > y = scipy.io.loadmat("x.mat", squeeze_me=True)
> > # print y
> > cd = y['CanData']
> > msg = cd['msg']
> > print msg
> > print msg.dtype
> > print msg.dtype.names
> >
> > I get output:
> > >C:\Anaconda2\pythonw -u "test1.py"
> > ((array(((array([ 61.96, 61.96, 61.96]), u'PosAct'), (array([-0.05,
> -0.1 , 0.3 ]), u'VelAct')),
> > dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0. ,
> 0.003968, 0.007978])),)
> > object
> > None
> > >Exit code: 0
> >
> > All well, but the name 'RPDO2' disapeared from the data!
> >
> > Now I need this information; in future I won't handle what's put into
> x.mat, so I need a way to access through the data all the way down (and
> handle the variations that will come).
> >
> > I have found a workaround at:
> > http://stackoverflow.com/questions/7008608/scipy-io-
> loadmat-nested-structures-i-e-dictionaries/ <http://stackoverflow.com/
> questions/7008608/scipy-io-loadmat-nested-structures-i-e-dictionaries/>
> >
> > The problem is, the workaround uses struct_as_record=False in loadmat,
> and which boils down to using scipy.io.matlab.mio5_params.mat_struct, and
> when you read the docstring of class mat_struct, it says:
> >
> > '''
> > ...
> > We deprecate this method of holding struct information, and will
> > soon remove it, in favor of the recarray method (see loadmat
> > docstring)
> > '''
> > So my questions:
> > 1) Did I miss something? Is there a way to access the data in 'RPDO2' by
> using this name, without using parameter struct_as_record=False in loadmat?
> > 2) If not, where do I file a bug? The workaround is five years old, so
> the issue seems to be in scipy for ages...
> >
> > (For the records, I use scipy within Anaconda2 1.4.1, under Windows, but
> this does not seem to matter).
> >
> > Thanks a lot for the answers, in advance.
> >
> > Nenad
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at python.org
> > https://mail.python.org/mailman/listinfo/scipy-user
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-user/
> attachments/20170222/8be8fa5c/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 22 Feb 2017 09:11:16 -0700
> From: Jason Sachs <jmsachs at gmail.com>
> To: SciPy Users List <scipy-user at python.org>
> Subject: Re: [SciPy-User] How to handle a scipy.io.loadmat - related
> bug: parts of the data inaccessible after loadmat
> Message-ID:
> <CAOo6sON=nTaf1_zp_4Ot7ibH+i7=du+p_R=3_hcOAtLrkaP8Rg at mail.
> gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> This looks familiar, I ran into this a few years ago, and if I recall
> correctly, there is an option to loadmat to reduce array dimensions
> appropriately. There is a "squeeze_me" option (unfortunately named...
> should probably be deprecated in favor of "squeeze") which I think does
> this.
>
> https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html
>
> On Wed, Feb 22, 2017 at 9:02 AM, Gregor Thalhammer <
> gregor.thalhammer at gmail.com> wrote:
>
> >
> > Am 22.02.2017 um 12:02 schrieb Propadovic Nenad <npropadovic at gmail.com>:
> >
> > Hello,
> >
> > bear with me for the long post that follows: it took me more than a week
> > to get this far, and I tried to compress all the relevant information
> into
> > the post.
> >
> > There seems to be a bug in scipy.io.loadmat; I'll present it by a short
> > piece of code and it's output.
> >
> > I create file x.mat with the following:
> >
> > import scipy.io
> >
> > d = {'CanData':
> > {
> > 'msg': {
> > 'RPDO2': {
> > 'timest': [0.0, 0.0039679999899817631,
> > 0.0079779999941820279],
> > 'sig': {
> > 'VelAct': {
> > 'Values': [-0.050000000000000003,
> > -0.10000000000000001, 0.29999999999999999, ],
> > 'Name': 'VelAct'
> > },
> > 'PosAct': {
> > 'Values': [61.960000000000001,
> 61.960000000000001,
> > 61.960000000000001, ],
> > 'Name': 'PosAct'
> > }
> > }
> > }
> > }
> > }
> > }
> > scipy.io.savemat("x.mat", d)
> >
> > Matlab is happy with the file and handles it the way I expect.
> >
> > When I read in the data stored in the file and print it out:
> >
> > import scipy.io
> > y = scipy.io.loadmat("x.mat")
> > # print y
> > cd = y['CanData']
> > msg = cd['msg']
> > print msg
> > print msg.dtype
> > print msg.dtype.names
> >
> > The output is:
> > >C:\Anaconda2\pythonw -u "test1.py"
> > [[ array([[ ([[(array([[ ([[(array([[ 61.96, 61.96, 61.96]]),
> > array([u'PosAct'],
> > dtype='<U6'))]], [[(array([[-0.05, -0.1 , 0.3 ]]),
> > array([u'VelAct'],
> > dtype='<U6'))]])]],
> > dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([[ 0. ,
> > 0.003968, 0.007978]]))]],)]],
> > dtype=[('RPDO2', 'O')])]]
> > object
> > None
> >
> > Now I've read the manual, and as I see it I have no way for me to access
> > the deeper layers of data I just put in the file x.mat, although they are
> > obviously right there in the data read in. Access via msg['RPDO2'] gives:
> > IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis
> > (`None`) and integer or boolean arrays are valid indices.
> >
> >
> > For historic reasons, in Matlab everything is at least a 2D array, even
> > scalars. By sprinkling some [0,0] in your code you should get what you
> > want, e.g.
> >
> > msg[0,0]['RPDO2'][0,0]['timest'][0,0]
> >
> > array([[ 0. , 0.003968, 0.007978]])
> >
> >
> > Gregor
> >
> >
> >
> > If I use parameter squeeze_me=True:
> >
> > scipy.io.savemat("x.mat", d)
> > y = scipy.io.loadmat("x.mat", squeeze_me=True)
> > # print y
> > cd = y['CanData']
> > msg = cd['msg']
> > print msg
> > print msg.dtype
> > print msg.dtype.names
> >
> > I get output:
> > >C:\Anaconda2\pythonw -u "test1.py"
> > ((array(((array([ 61.96, 61.96, 61.96]), u'PosAct'), (array([-0.05,
> -0.1
> > , 0.3 ]), u'VelAct')),
> > dtype=[('PosAct', 'O'), ('VelAct', 'O')]), array([ 0. ,
> > 0.003968, 0.007978])),)
> > object
> > None
> > >Exit code: 0
> >
> > All well, but the name 'RPDO2' disapeared from the data!
> >
> > Now I need this information; in future I won't handle what's put into
> > x.mat, so I need a way to access through the data all the way down (and
> > handle the variations that will come).
> >
> > I have found a workaround at:
> > http://stackoverflow.com/questions/7008608/scipy-io-
> > loadmat-nested-structures-i-e-dictionaries/
> >
> > The problem is, the workaround uses struct_as_record=False in loadmat,
> and
> > which boils down to using scipy.io.matlab.mio5_params.mat_struct, and
> > when you read the docstring of class mat_struct, it says:
> >
> > '''
> > ...
> > We deprecate this method of holding struct information, and will
> > soon remove it, in favor of the recarray method (see loadmat
> > docstring)
> > '''
> > So my questions:
> > 1) Did I miss something? Is there a way to access the data in 'RPDO2' by
> > using this name, without using parameter struct_as_record=False in
> loadmat?
> > 2) If not, where do I file a bug? The workaround is five years old, so
> the
> > issue seems to be in scipy for ages...
> >
> > (For the records, I use scipy within Anaconda2 1.4.1, under Windows, but
> > this does not seem to matter).
> >
> > Thanks a lot for the answers, in advance.
> >
> > Nenad
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at python.org
> > https://mail.python.org/mailman/listinfo/scipy-user
> >
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at python.org
> > https://mail.python.org/mailman/listinfo/scipy-user
> >
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/scipy-user/
> attachments/20170222/a4113f89/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at python.org
> https://mail.python.org/mailman/listinfo/scipy-user
>
>
> ------------------------------
>
> End of SciPy-User Digest, Vol 162, Issue 5
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-user/attachments/20170223/79b020b4/attachment-0001.html>
More information about the SciPy-User
mailing list