[Numpy-discussion] Switch to using ATLAS for OSX binary wheels
Matthew Brett
matthew.brett at gmail.com
Mon Jun 16 07:10:47 EDT 2014
Hi,
On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>
>
>
> On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>>
>>
>> On Friday, June 13, 2014, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>
>>>
>>>
>>>
>>> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett at gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Summary : I'm planning to upload OSX wheels for numpy and scipy using
>>>> the ATLAS blas / lapack library instead of the default OSX Accelerate
>>>> framework.
>>>>
>>>> We've run into some trouble with a segfault in recent OSX Accelerate:
>>>>
>>>> https://github.com/numpy/numpy/issues/4007
>>>>
>>>> and Accelerate also doesn't play well with multiprocessing.
>>>>
>>>> https://github.com/numpy/numpy/issues/4776
>>>>
>>>> Because there's nothing I love more than half-day compilation runs on
>>>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy
>>>> and scipy to them to make OSX wheels. These pass all tests in i386
>>>> and x86_64 mode, including numpy, scipy, matplotlib, pandas:
>>>>
>>>>
>>>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
>>>>
>>>> The build process needs some automating, but it's recorded here:
>>>>
>>>> https://github.com/matthew-brett/numpy-atlas-binaries
>>>>
>>>> It's possible to get travis-ci to build these guys from a bare machine
>>>> and then upload them somewhere, but I haven't tried to do that.
>>>>
>>>> Meanwhile Sturla kindly worked up a patch to numpy to work round the
>>>> Accelerate segfault [1]. I haven't tested that, but given I'd already
>>>> built the wheels, I prefer the ATLAS builds because they work with
>>>> multiprocessing.
>>>>
>>>> I propose uploading these wheels as the default for numpy and scipy.
>>>> Does anyone have any objection or comments before I go ahead and do
>>>> that?
>>>
>>>
>>> From your README and wscript I don't see what numpy version you're using
>>> to compile scipy against. I got the impression that you used 1.8.1, but it
>>> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
>>>
>>> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see
>>> below). Your wheels should work with all common Python installs (mine is
>>> homebrew) right?
>>>
>>> Ralf
>>>
>>>
>>> $ python2.7 -c "import scipy; scipy.test()"
>>> Running unit tests for scipy
>>> NumPy version 1.9.0.dev-056ab73
>>> NumPy is installed in
>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy
>>> SciPy version 0.14.0
>>> SciPy is installed in
>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy
>>> Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1
>>> Compatible Apple LLVM 4.2 (clang-425.0.28)]
>>> nose version 1.3.0
>>> E...............EEEEEE............EEEEEEEEEE
>>>
>>> ======================================================================
>>> ERROR: Failure: ImportError
>>> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so,
>>> 2): Symbol not found: _PyModule_Create2
>>> Referenced from:
>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
>>> Expected in: flat namespace
>>> in
>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so)
>>>
>>> ----------------------------------------------------------------------
>>> Traceback (most recent call last):
>>> File
>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py",
>>> line 413, in loadTestsFromName
>>> addr.filename, addr.module)
>>> File
>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py",
>>> line 47, in importFromPath
>>> return self.importFromDir(dir_path, fqname)
>>> File
>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py",
>>> line 94, in importFromDir
>>> mod = load_module(part_fqname, fh, filename, desc)
>>> File
>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py",
>>> line 27, in <module>
>>> from . import vq, hierarchy
>>> File
>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py",
>>> line 175, in <module>
>>> from . import _hierarchy_wrap
>>> ImportError:
>>> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so,
>>> 2): Symbol not found: _PyModule_Create2
>>> Referenced from:
>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
>>> Expected in: flat namespace
>>> in
>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
>>>
>>> ...
>>> <42 more errors>
>>
>>
>> That is strange homebrew is one of tests in the grid and the installation
>> path looks strange.
>>
>> Can you try downloading the wheel from the url and installing from the
>> local file?
>
>
> That's what I did (easier than remembering the magic pip incantation). The
> install path looks fine to me, maybe homebrew changed it recently? I can try
> to update my install, will take a few days though.
Yes, sorry - that does look like the normal homebrew install path, I
didn't realize it had the exotic framework parts to it.
I just ran these commands on my machine:
SPI=https://nipy.bic.berkeley.edu/scipy_installers
BREW_BIN=/usr/local/Cellar/python/2.7.6/bin
curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
$BREW_BIN/pip install
numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
$BREW_BIN/pip install
scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
$BREW_BIN/python -c "import scipy; scipy.test()"
and the scipy tests passed.
I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies
binary compatible? What difference would I expect to see if I'd built
scipy against numpy 1.5.1 or 1.7 ?
Cheers,
Matthew
More information about the NumPy-Discussion
mailing list