[Numpy-discussion] Switch to using ATLAS for OSX binary wheels

Matthew Brett matthew.brett at gmail.com
Thu Jun 19 13:16:27 EDT 2014 

On Thu, Jun 19, 2014 at 11:39 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Mon, Jun 16, 2014 at 1:00 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On Mon, Jun 16, 2014 at 12:51 PM,  <josef.pktd at gmail.com> wrote:
>>> On Mon, Jun 16, 2014 at 7:10 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>>>> Hi,
>>>>
>>>> On Sun, Jun 15, 2014 at 10:51 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Jun 14, 2014 at 11:56 AM, Matthew Brett <matthew.brett at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Friday, June 13, 2014, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 13, 2014 at 2:07 PM, Matthew Brett <matthew.brett at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Summary : I'm planning to upload OSX wheels for numpy and scipy using
>>>>>>>> the ATLAS blas / lapack library instead of the default OSX Accelerate
>>>>>>>> framework.
>>>>>>>>
>>>>>>>> We've run into some trouble with a segfault in recent OSX Accelerate:
>>>>>>>>
>>>>>>>> https://github.com/numpy/numpy/issues/4007
>>>>>>>>
>>>>>>>> and Accelerate also doesn't play well with multiprocessing.
>>>>>>>>
>>>>>>>> https://github.com/numpy/numpy/issues/4776
>>>>>>>>
>>>>>>>> Because there's nothing I love more than half-day compilation runs on
>>>>>>>> my laptop, I've built ATLAS binaries with gcc 4.8, and linked numpy
>>>>>>>> and scipy to them to make OSX wheels.  These pass all tests in i386
>>>>>>>> and x86_64 mode, including numpy, scipy, matplotlib, pandas:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/27442987
>>>>>>>>
>>>>>>>> The build process needs some automating, but it's recorded here:
>>>>>>>>
>>>>>>>> https://github.com/matthew-brett/numpy-atlas-binaries
>>>>>>>>
>>>>>>>> It's possible to get travis-ci to build these guys from a bare machine
>>>>>>>> and then upload them somewhere, but I haven't tried to do that.
>>>>>>>>
>>>>>>>> Meanwhile Sturla kindly worked up a patch to numpy to work round the
>>>>>>>> Accelerate segfault [1].  I haven't tested that, but given I'd already
>>>>>>>> built the wheels, I prefer the ATLAS builds because they work with
>>>>>>>> multiprocessing.
>>>>>>>>
>>>>>>>> I propose uploading these wheels as the default for numpy and scipy.
>>>>>>>> Does anyone have any objection or comments before I go ahead and do
>>>>>>>> that?
>>>>>>>
>>>>>>>
>>>>>>> From your README and wscript I don't see what numpy version you're using
>>>>>>> to compile scipy against. I got the impression that you used 1.8.1, but it
>>>>>>> should be numpy 1.5.1 for the 2.7 build, and 1.7.1 for 3.x.
>>>>>>>
>>>>>>> I've tried the scipy 0.14.0 python2.7 wheel, but I get import errors (see
>>>>>>> below). Your wheels should work with all common Python installs (mine is
>>>>>>> homebrew) right?
>>>>>>>
>>>>>>> Ralf
>>>>>>>
>>>>>>>
>>>>>>>     $ python2.7 -c "import scipy; scipy.test()"
>>>>>>>     Running unit tests for scipy
>>>>>>>     NumPy version 1.9.0.dev-056ab73
>>>>>>>     NumPy is installed in
>>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy
>>>>>>>     SciPy version 0.14.0
>>>>>>>     SciPy is installed in
>>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy
>>>>>>>     Python version 2.7.5 (default, Jun 18 2013, 21:21:44) [GCC 4.2.1
>>>>>>> Compatible Apple LLVM 4.2 (clang-425.0.28)]
>>>>>>>     nose version 1.3.0
>>>>>>>     E...............EEEEEE............EEEEEEEEEE
>>>>>>>
>>>>>>> ======================================================================
>>>>>>>     ERROR: Failure: ImportError
>>>>>>> (dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so,
>>>>>>> 2): Symbol not found: _PyModule_Create2
>>>>>>>       Referenced from:
>>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
>>>>>>>       Expected in: flat namespace
>>>>>>>      in
>>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so)
>>>>>>>
>>>>>>> ----------------------------------------------------------------------
>>>>>>>     Traceback (most recent call last):
>>>>>>>       File
>>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/loader.py",
>>>>>>> line 413, in loadTestsFromName
>>>>>>>         addr.filename, addr.module)
>>>>>>>       File
>>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py",
>>>>>>> line 47, in importFromPath
>>>>>>>         return self.importFromDir(dir_path, fqname)
>>>>>>>       File
>>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/importer.py",
>>>>>>> line 94, in importFromDir
>>>>>>>         mod = load_module(part_fqname, fh, filename, desc)
>>>>>>>       File
>>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/__init__.py",
>>>>>>> line 27, in <module>
>>>>>>>         from . import vq, hierarchy
>>>>>>>       File
>>>>>>> "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.py",
>>>>>>> line 175, in <module>
>>>>>>>         from . import _hierarchy_wrap
>>>>>>>     ImportError:
>>>>>>> dlopen(/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so,
>>>>>>> 2): Symbol not found: _PyModule_Create2
>>>>>>>       Referenced from:
>>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
>>>>>>>       Expected in: flat namespace
>>>>>>>      in
>>>>>>> /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/_hierarchy_wrap.so
>>>>>>>
>>>>>>> ...
>>>>>>> <42 more errors>
>>>>>>
>>>>>>
>>>>>> That is strange homebrew is one of tests in the grid and the installation
>>>>>> path looks strange.
>>>>>>
>>>>>> Can you try downloading the wheel from the url and installing from the
>>>>>> local file?
>>>>>
>>>>>
>>>>> That's what I did (easier than remembering the magic pip incantation). The
>>>>> install path looks fine to me, maybe homebrew changed it recently? I can try
>>>>> to update my install, will take a few days though.
>>>>
>>>> Yes, sorry - that does look like the  normal homebrew install path, I
>>>> didn't realize it had the exotic framework parts to it.
>>>>
>>>> I just ran these commands on my machine:
>>>>
>>>> SPI=https://nipy.bic.berkeley.edu/scipy_installers
>>>> BREW_BIN=/usr/local/Cellar/python/2.7.6/bin
>>>> curl -O $SPI/numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
>>>> curl -O $SPI/scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
>>>> $BREW_BIN/pip install
>>>> numpy-1.8.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
>>>> $BREW_BIN/pip install
>>>> scipy-0.14.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
>>>> $BREW_BIN/python -c  "import scipy; scipy.test()"
>>>>
>>>> and the scipy tests passed.
>>>>
>>>> I built the scipy wheel against numpy 1.8.1 - but - aren't the numpies
>>>> binary compatible?  What difference would I expect to see if I'd built
>>>> scipy against numpy 1.5.1 or 1.7 ?
>>>
>>> I summarized here what David explained a while ago about the
>>> difference between forward and backwards binary compatibility
>>>
>>> http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312
>>>
>>
>> I see - thanks for the summary.  I will recompile.
>
> Last call - binaries recompiled, tested, more comments here:
>
> https://github.com/numpy/numpy/issues/4007
>
> I'm planning to upload these binaries to pypi later on today,

Done - please test and let me know of any problems,

Cheers,

Matthew

More information about the NumPy-Discussion mailing list