[Neuroimaging] Using Python libs from MATLAB

[Marmaduke Woodman]

hi

To respond to the different open points / questions in previous emails:

> Ouch - I guess this means it's not possible to use h5py or pytables?

It is, but you have to compile them with a version of HDF5 includes that
match MATLAB's, so that the API matches.

> what exactly do you mean "anything touching HDF5"? What series of steps
causes this segfault?

In fact, the h5py/pytables modules load symbols from the HDF5 lib, which as
a first sanity check, verifies that the header version matches the library
version. If not, abort(). MATLAB uses an older version of HDF5 than
Anaconda's defaults hence the problem. Compiling these libs with a header
matching MATLAB's version works works OK.

> anything that is matlab >= 2006b

They call it mat file version 7.3. Version 7 or less is fine, libmatio (and
scipy) can handle it.

> I don't think Mathworks has documented their 7.3 format.

Yep, IIRC they put weird stuff in the HDF5 which isn't trivially readable
by h5py, but Octave people must have reverse engineered it, so would be a
good reference point.

> What operating systems have you tested these on?  I wouldn't be surprised
if you found that all of the above issues happened only on Linux

Windows, Mac & Linux. My experience is that toolchains, compiler flags and
library versions are more consistent on Windows & Mac, leading to fewer
surprises.

> Too bad about lin alg

The main issue is that MKL used by MATLAB uses the same symbol names but
not the same ABI, so NumPy's extension modules' references resolve to MKL
symbols and not its own and put the args on the stack in the wrong order.
Monkey patching around it would be not difficult by intercepting
numpy.linalg calls and routing them to MATLAB's MEX Lapack routines via
ctypes.

It would also be possible to build NumPy against MATLAB's MKL BLAS & LAPACK
routines, though my suggestion on the numpy mailing list about this got no
replies.

Anaconda with MKL though appears to work fine, since the relevant Python
modules (numpy..) are compiled against MKL in that case.

>> - Scipy.io.{save,load}mat segfault
> That's interesting - it's probably possible to fix that, because that stuff
doesn't call any external libraries.  Do you have any more
details?

I thought I had seen a symbol in the stacktrace pointing to libmatio I
believe, but I just read mio5.py and it's all in Python, so I'll have to go
back and check, maybe it was a h5py load.

> I wish that they told us what they are doing (rather than providing an
opaque executable).

To be fair, it was in their release notes ;)



Cheers,
Marmaduke

On Mon, Jun 13, 2016 at 9:10 PM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> Hi Dimitri,
>
> On Mon, Jun 13, 2016 at 11:15 AM, Dimitri Papadopoulos Orfanos
> <dimitri.papadopoulos at cea.fr> wrote:
> > Hi Matthew,
> >
> > There are a few links referring to the Matlab 7.3 format:
> >
> http://scipy-cookbook.readthedocs.io/items/Reading_mat_files.html#matlab-7-3-and-greater
>
> Yes, it's not hard to open the files with h5py or similar, the work
> will be to see how the file structure relates to the saved variables.
> It's quite possible that it is fairly easy to work out - I haven't
> looked.
>
> >
> http://stackoverflow.com/questions/4950630/matlab-differences-between-mat-versions
> >
> > There is also this PDF from Matlab which does not contain even once the
> > "HDF" string but does seem to refer to the HDF5-based 7.3 format:
> > http://www.mathworks.com/help/pdf_doc/matlab/matfile_format.pdf
>
> That's the doc that describes the earlier mat file formats (versions 4
> and 5).   scipy.io implements all the stuff described in that doc.
> Quoting from the introduction there:
>
> "This document describes the internal format of MATLAB Level 4 and
> Level 5 MAT files."
>
> See you,
>
> Matthew
> _______________________________________________
> Neuroimaging mailing list
> Neuroimaging at python.org
> https://mail.python.org/mailman/listinfo/neuroimaging
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/neuroimaging/attachments/20160614/aae6c826/attachment.html>