[SciPy-Dev] runtest issue mac os with master 14de485c053cc275682bc7de0316fcca72b341f8

Tue Jun 5 19:27:00 EDT 2018

On Tue, Jun 5, 2018 at 9:17 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Tue, Jun 5, 2018 at 8:09 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>> On Tue, Jun 5, 2018 at 7:30 PM, Ralf Gommers <ralf.gommers at gmail.com> wrote:
>>>
>>>
>>> On Tue, Jun 5, 2018 at 11:14 AM, Matthew Brett <matthew.brett at gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Sat, Jun 2, 2018 at 6:05 PM, Ralf Gommers <ralf.gommers at gmail.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > On Sat, Jun 2, 2018 at 9:57 AM, Mark Alexander Mikofski
>>>> > <mikofski at berkeley.edu> wrote:
>>>> >>
>>>> >> Hi All,
>>>> >>
>>>> >> I am having issues using
>>>> >>
>>>> >>     python runtest.py
>>>> >>
>>>> >> from scipy master branch since commit
>>>> >> 14de485c053cc275682bc7de0316fcca72b341f8
>>>> >>
>>>> >>
>>>> >> https://github.com/scipy/scipy/tree/14de485c053cc275682bc7de0316fcca72b341f8
>>>> >>
>>>> >> there are two tracebacks, I think this may have something to do with
>>>> >> changes to LAPACK and dropping Apple Accelerate support, but not sure,
>>>> >> and
>>>> >> don't know how to fix it. Should I brew install openblas? (I'm going to
>>>> >> try
>>>> >> that)
>>>> >
>>>> >
>>>> > Yes that is the recommendation (if you're not using Anaconda).
>>>> >
>>>> >
>>>> >>
>>>> >> Use anaconda and create a conda env? (maybe this next?)
>>>> >
>>>> >
>>>> > If you're already using Anaconda, and the numpy from the defaults
>>>> > channel,
>>>> > SciPy should automatically link against MKL.
>>>> >
>>>> >
>>>> >>
>>>> >> Currently I'm using python 3.6 from homebrew in a virtualenv with
>>>> >> wheels
>>>> >> from PyPI, after I create the virtualenv using the homebrew python-3
>>>> >> interpreter, I do pip install requirements.txt (see attached).
>>>> >>
>>>> >> 1st traceback:
>>>> >>
>>>> >> ImportError while importing test module
>>>> >>
>>>> >> '/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/cluster/tests/test_hierarchy.py'.
>>>> >> Hint: make sure your test modules/packages have valid Python names.
>>>> >> Traceback:
>>>> >> scipy/cluster/__init__.py:27: in <module>
>>>> >>     from . import vq, hierarchy
>>>> >> scipy/cluster/vq.py:75: in <module>
>>>> >>     from scipy.spatial.distance import cdist
>>>> >> scipy/spatial/__init__.py:97: in <module>
>>>> >>     from ._spherical_voronoi import SphericalVoronoi
>>>> >> scipy/spatial/_spherical_voronoi.py:19: in <module>
>>>> >>     from scipy.spatial.distance import pdist
>>>> >> scipy/spatial/distance.py:123: in <module>
>>>> >>     from ..linalg import norm
>>>> >> scipy/linalg/__init__.py:207: in <module>
>>>> >>     from ._decomp_update import *
>>>> >> _decomp_update.pyx:1: in init scipy.linalg._decomp_update
>>>> >>     ???
>>>> >> E   ImportError:
>>>> >>
>>>> >> dlopen(/Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so,
>>>> >> 2): Symbol not found: _cbbcsd_
>>>> >> E     Referenced from:
>>>> >>
>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>> >> E     Expected in: flat namespace
>>>> >> E    in
>>>> >>
>>>> >> /Users/markmikofski/Projects/scipy/build/testenv/lib/python3.6/site-packages/scipy/linalg/cython_lapack.cpython-36m-darwin.so
>>>>
>>>> I am getting the same error, for OpenBLAS v0.3.0, on the Travis-CI
>>>> macOS machines, using our usual wheel-building procedure (+OpenBLAS):
>>>>
>>>> * Python 3.4:
>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298604
>>>> * Python 3.5:
>>>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298605
>>>
>>>
>>> You're cloning OpenBLAS 0.3.0 but it's not detected (you'll either need a
>>> site.cfg or install it to one of the regular locations):
>>>
>>> openblas_info:
>>>
>>>   libraries  not found in
>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>> '/usr/local/lib', '/usr/lib']
>>>
>>>   NOT AVAILABLE
>>>
>>>
>>>
>>> atlas_blas_threads_info:
>>>
>>> Setting PTATLAS=ATLAS
>>>
>>>   libraries ptf77blas,ptcblas,atlas not found in
>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>> '/usr/local/lib', '/usr/lib']
>>>
>>>   NOT AVAILABLE
>>>
>>>
>>>
>>> atlas_blas_info:
>>>
>>>   libraries f77blas,cblas,atlas not found in
>>> ['/Users/travis/build/matthew-brett/scipy-wheels/venv/bin/../lib',
>>> '/usr/local/lib', '/usr/lib']
>>>
>>>   NOT AVAILABLE
>>>
>>>
>>>
>>>   FOUND:
>>>
>>>     libraries = ['blas']
>>>
>>>     library_dirs = ['/usr/lib']
>>>
>>>     define_macros = [('NO_ATLAS_INFO', 1)]
>>>
>>>     language = f77
>>
>> Hmm - you're right - that's very odd.  OpenBLAS is being unpacked into
>> /usr/local/lib, where it should be detected.  On Python 3.6, using the
>> same machinery, it is detected - see:
>>
>> https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388298606
>
> It looks like it's numpy distutils.  numpy 1.8.2 doesn't appear to find
> OpenBLAS in /usr/local/lib, even though it claims it looked.  Numpy
> 1.9.3 does find it, which is why the build is OK for Python 3.6.
> Writing an explicit site.cfg:
>
>     cat << EOF > $HOME/site.cfg
> [openblas]
> libraries = openblas
> library_dirs = /usr/local/lib
> include_dirs = /usr/local/include
> runtime_library_dirs = /usr/local/lib
> EOF
>
> fixes that locally.  Testing on Travis-CI now:
>
> https://travis-ci.org/matthew-brett/scipy-wheels/builds/388461944

Oops, no, Python 3.6 was using numpy 1.11.3, but this is now broken
with the site.cfg, giving

ld: unknown option: -rpath=/usr/local/lib

https://travis-ci.org/matthew-brett/scipy-wheels/jobs/388461965#L8489

I can replicate this locally.  Deleting site.cfg fixes the error.

The other builds continue to be broken with the same kind of errors;
they pick up OpenBLAS, but also the /usr/lib stuff.  So, you're right,
I think it needs blacklisting.  I hope that would make it possible to
build with numpy 1.9.3 at least, making sure to avoid having a
site.cfg.

Cheers,

Matthew