[Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

[Olivier Grisel]

Hi Carl,

I installed Python 2.7.6 64 bits on a windows server instance from
rackspace cloud and then ran get-pip.py and then could successfully
install the numpy and scipy wheel packages from your google drive
folder. I tested dot products and scipy.linalg.svd and they work as
expected.

Then I uncompressed your mingw toolchain in c:\mingw, put c:\mingw\bin
in my PATH and tried to build the scikit-learn git master with it,
however it fails with:

building 'sklearn.__check_build._check_build' extension
compiling C sources
C compiler: gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes

compile options: '-D__MSVCRT_VERSION__=0x0900
-Ic:\Python27\lib\site-packages\numpy\core\include
-Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python2
7\include -Ic:\Python27\PC -c'
gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes
-D__MSVCRT_VERSION__=0x0900
-Ic:\Python27\lib\site-packages\numpy\core\include
-Ic:\Python27\lib\site-
packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c
sklearn\__check_build\_check_build.c -o
build\temp.win-amd64-2.7\Release\sklearn\__check_b
uild\_check_build.o
Found executable c:\mingw\bin\gcc.exe
gcc -shared -Wl,-gc-sections -Wl,-s
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o
-Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -Lbuild
\temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o
build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x3):
undefined reference to `__imp__Py_NoneStruct'
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x1ca):
undefined reference to `__imp__PyThreadState_Current'
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x405):
undefined reference to `__imp_PyExc_ImportError'
c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.8.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.
o: bad reloc address 0x0 in section `.data'
collect2.exe: error: ld returned 1 exit status
error: Command "gcc -shared -Wl,-gc-sections -Wl,-s
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o
-Lc:\Python27\libs -Lc:\Python27\PCbui
ld\amd64 -Lbuild\temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o
build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd" failed
with exit status 1

Furthermore, when I try to introspect the blas information on this box I get:

In [1]: import scipy
C:\Python27\lib\site-packages\numpy\core\__init__.py:6: Warning: Numpy
64bit experimental build with Mingw-w64 and OpenBlas. Use with care.
  from . import multiarray
OpenBLAS : Your OS does not support AVX instructions. OpenBLAS is
using Barcelona kernels as a fallback, which may give poorer
performance.

In [2]: scipy.show_config()
umfpack_info:
  NOT AVAILABLE
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['D:/devel/mingw64static/x86_64-w64-mingw32/lib']
    language = f77
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['D:/devel/mingw64static/x86_64-w64-mingw32/lib']
    language = f77
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['D:/devel/mingw64static/x86_64-w64-mingw32/lib']
    language = f77
blas_mkl_info:
  NOT AVAILABLE

In [3]: from numpy.distutils.system_info import get_info

In [4]: get_info('blas_opt')
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:576:
UserWarning: Specified path
D:/devel/mingw64static/x86_64-w64-mingw32/lib is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:1522: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:1531: UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:1534: UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  warnings.warn(BlasSrcNotFoundError.__doc__)
Out[4]: {}

Would it make sense to embed the blas and lapack header files as part
of this numpy wheel and make numpy.distutils.system_info return the
lib and include folder pointing to the embedded libopenblas.dll and
header files so has to make third party libraries directly buildable
against those?

-- 
Olivier