[SciPy-Dev] runtest issue mac os with master 14de485c053cc275682bc7de0316fcca72b341f8

Wed Jun 6 06:55:09 EDT 2018

On Wed, Jun 6, 2018 at 12:27 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> Hi,
>>>
>>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>>
>>>>
>>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>>>> wrote:
>>>>> >
>>>>> >
>>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>>>>> > <mikofski at berkeley.edu> wrote:
>>>>> >>
>>>>> >> Hi All,
>>>>> >>
>>>>> >> I am having issues using
>>>>> >>
>>>>> >>     python runtest.py
>>>>> >>
>>>>> >> from scipy master branch since commit
>>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>>>>> >>
>>>>> >>
>>>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>>>>> >>
>>>>> >> there are two tracebacks, I think this may have something to do with
>>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>>>>> >> and
>>>>> >> don't know how to fix it. Should I brew install openblas? (I'm going to
>>>>> >> try
>>>>> >> that)
>>>>> >
>>>>> >
>>>>> > Yes that is the recommendation (if you're not using Anaconda).
>>>>> >
>>>>> >
>>>>> >>
>>>>> >> Use anaconda and create a conda env? (maybe this next?)
>>>>> >
>>>>> >
>>>>> > If you're already using Anaconda, and the numpy from the defaults
>>>>> > channel,
>>>>> > SciPy should automatically link against MKL.
>>>>> >
>>>>> >
>>>>> >>
>>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>>>>> >> wheels
>>>>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>>>>> >> interpreter, I do pip install requirements.txt (see attached).
>>>>> >>
>>>>> >> 1st traceback:
>>>>> >>
>>>>> >> ImportError while importing test module
>>>>> >>
>>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>>>>> >> Hint: make sure your test modules/packages have valid Python names.
>>>>> >> Traceback:
>>>>> >> scipy/cluster/__init__.py:27: in <module>
>>>>> >>     from . import vq, hierarchy
>>>>> >> scipy/cluster/vq.py:75: in <module>
>>>>> >>     from scipy.spatial.distance import cdist
>>>>> >> scipy/spatial/__init__.py:97: in <module>
>>>>> >>     from ._spherical_voronoi import SphericalVoronoi
>>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>>>>> >>     from scipy.spatial.distance import pdist
>>>>> >> scipy/spatial/distance.py:123: in <module>
>>>>> >>     from ..linalg import norm
>>>>> >> scipy/linalg/__init__.py:207: in <module>
>>>>> >>     from ._decomp_update import *
>>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>>>>> >>     ???
>>>>> >> E   ImportError:
>>>>> >>
>>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>>>>> >> 2): Symbol not found: _cbbcsd_
>>>>> >> E     Referenced from:
>>>>> >>
>>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>>> >> E     Expected in: flat namespace
>>>>> >> E    in
>>>>> >>
>>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>>>
>>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>>>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>>>>
>>>>> * Python 3.4:
>>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>>>>> * Python 3.5:
>>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>>>>
>>>>
>>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
>>>> site.cfg or install it to one of the regular locations):
>>>>
>>>> openblas_info:
>>>>
>>>>   libraries  not found in
>>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>>> '/usr/local/lib', '/usr/lib']
>>>>
>>>>   NOT AVAILABLE
>>>>
>>>>
>>>>
>>>> atlas_blas_threads_info:
>>>>
>>>> Setting PTATLAS=ATLAS
>>>>
>>>>   libraries ptf77blas,ptcblas,atlas not found in
>>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>>> '/usr/local/lib', '/usr/lib']
>>>>
>>>>   NOT AVAILABLE
>>>>
>>>>
>>>>
>>>> atlas_blas_info:
>>>>
>>>>   libraries f77blas,cblas,atlas not found in
>>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>>> '/usr/local/lib', '/usr/lib']
>>>>
>>>>   NOT AVAILABLE
>>>>
>>>>
>>>>
>>>>   FOUND:
>>>>
>>>>     libraries = ['blas']
>>>>
>>>>     library_dirs = ['/usr/lib']
>>>>
>>>>     define_macros = [('NO_ATLAS_INFO', 1)]
>>>>
>>>>     language = f77
>>>
>>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
>>> /usr/local/lib, where it should be detected.  On Python 3.6, using the
>>> same machinery, it is detected - see:
>>>
>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
>>
>> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
>> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
>> 1.9.3 does find it, which is why the build is OK for Python 3.6.
>> Writing an explicit site.cfg:
>>
>>     cat << EOF > $HOME/site.cfg
>> [openblas]
>> libraries = openblas
>> library_dirs = /usr/local/lib
>> include_dirs = /usr/local/include
>> runtime_library_dirs = /usr/local/lib
>> EOF
>>
>> fixes that locally.  Testing on Travis-CI now:
>>
>> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944
>
> Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
> with the site.cfg, giving
>
> ld: unknown option: -rpath=/usr/local/lib
>
> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489
>
> I can replicate this locally.  Deleting site.cfg fixes the error.
>
> The other builds continue to be broken with the same kind of errors;
> they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
> I think it needs blacklisting.  I hope that would make it possible to
> build with numpy 1.9.3 at least, making sure to avoid having a
> site.cfg.

All macOS builds work correctly if I force numpy 1.11.3.

https://travis-ci.org/matthew-brett/scipy-wheels/builds/388525456

Any ideas why numpy 1.11.3 is not picking up the /usr/lib stuff?

Building wheels against 1.11.3 would be a fairly significant leap in
numpy requirement, so it would be better if we could build with numpy
1.9.3, at least.

Cheers,

Matthew