[SciPy-user] Building Python, Numpy and Scipy with Intel compilers and MKL

Tue Apr 28 22:29:23 EDT 2009

I spoke too soon!  Upon further testing of Numpy, I find that I cannot import the "sparsetools" package, which is built using Swig (details of the error below).  The system version of Swig is 1.3.29, built with g++.  Is it possible that Swig is causing a problem here?  Any other suggestions?

>>> from scipy.interpolate import interp1d
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/me/Python_Test/lib/python/scipy/interpolate/__init__.py", line
13, in <module>
    from rbf import Rbf
  File "/home/me/Python_Test/lib/python/scipy/interpolate/rbf.py", line 47, in <module>
    from scipy import linalg
  File "/home/me/Python_Test/lib/python/scipy/linalg/__init__.py", line 13, in <module>
    from iterative import *
  File "/home/me/Python_Test/lib/python/scipy/linalg/iterative.py", line 5, in <module>
    from scipy.sparse.linalg import isolve
  File "/home/me/Python_Test/lib/python/scipy/sparse/__init__.py", line 6, in <module>
    from csr import *
  File "/home/me/Python_Test/lib/python/scipy/sparse/csr.py", line 12, in <module>
    from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \
  File "/home/me/Python_Test/lib/python/scipy/sparse/sparsetools/__init__.py", line 4, in <module>
    from csr import *
  File "/home/me/Python_Test/lib/python/scipy/sparse/sparsetools/csr.py", line 7, in <module>
    import _csr
ImportError: /home/me/Python_Test/lib/python/scipy/sparse/sparsetools/_csr.so: undefined symbol: _ZNSt8ios_base4InitD1Ev

I looked at the symbols defined in the library with "nm" and sure enough, that one is undefined:
...
00000000000d2fdc W _ZNSt6vectorIySaIyEE6resizeEmy
                 U _ZNSt8ios_base4InitC1Ev
                 U _ZNSt8ios_base4InitD1Ev
...

For what it's worth, here is the build output for the building of _csr.so:
------------------------------------------------------------------------------------------
building 'scipy.sparse.sparsetools._csr' extension
compiling C++ sources
C compiler: icc -O2 -fPIC

creating build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools
compile options: '-I/home/cfinch/lib/python/numpy/core/include -I/home/cfinch/Python_Test/include/python2.5 -c'
icc: scipy/sparse/sparsetools/csr_wrap.cxx
scipy/sparse/sparsetools/csr_wrap.cxx(11526): (col. 28) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(11572): (col. 28) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10056): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10182): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10308): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10434): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10560): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10686): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10812): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(10938): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(11064): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(11190): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(11316): (col. 3) remark: LOOP WAS VECTORIZED.
scipy/sparse/sparsetools/csr_wrap.cxx(11442): (col. 3) remark: LOOP WAS VECTORIZED.
icc
-O2 -fPIC -shared
build/temp.linux-x86_64-2.5/scipy/sparse/sparsetools/csr_wrap.o
-Lbuild/temp.linux-x86_64-2.5 -o
build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_csr.so
...
copying
build/lib.linux-x86_64-2.5/scipy/sparse/sparsetools/_csr.so ->
/home/cfinch/Python_Test/lib/python/scipy/sparse/sparsetools

Any ideas?

            Craig

----- Original Message ----
From: Craig Finch <oanjao at yahoo.com>
To: scipy-user at scipy.org
Sent: Tuesday, April 28, 2009 2:56:22 PM
Subject: Re: Building Python, Numpy and Scipy with Intel compilers and MKL

I figured it out--I am now able to build Python, Numpy, and Scipy with the Intel C, C++, and Fortran compilers.  Here is the command line that I used:

/home/me/Python_Test/bin/python setup.py config --compiler=intel --fcompiler=intelem config_fc --fcompiler=intelem install --home=/home/me/Python_Test > scipy_build_log.txt

What I did differently to make it work: pass the --fcompiler=intelem option to the config_fc module. The option "--fcompiler=intel" did not work for me (perhaps because I am compiling on a 64-bit machine?)  Unfortunately, Numpy distutils are not well documented, so I had to read the source to figure this out.

Thanks to those who offered advice.  Now that I have two separate installs, one built with GCC and one with Intel, I will try to run some benchmarks to compare the two.

   Craig

----- Original Message ----
From: Craig Finch <oanjao at yahoo.com>
To: scipy-user at scipy.org
Sent: Wednesday, April 22, 2009 6:31:08 PM
Subject: Building Python, Numpy and Scipy with Intel compilers and MKL

I'm trying to build Python, Numpy, and SciPy on a Red Hat system using the Intel Fortran, C, and C++ compilers, along with the Intel Math Kernel Library.  This is proving to be difficult--I've tried everything I could find on the web, and it's still not working (everything works fine if I use GCC).  I would really appreciate any suggestions.

I can get Python and Numpy working correctly after compiling with Intel compilers, but I can't import anything from SciPy without getting an "undefined symbol" error.  According to what I've read, this can happen when gfortran is used to compile some library (such as BLAS) while Intel fortran is used to compile SciPy.  I don't think this is my problem, because I'm only using the Intel Math Kernel Libraries.  Is this correct?

Here's my latest attempt at installation:
Python install:
CC='icc'
CXX='icc' LDFLAGS='-L/opt/intel/cce/10.1.008/lib' CFLAGS='-I/opt/intel/cce/10.1.008/include' CXXFLAGS='-I/opt/intel/cce/10.1.008/include' ./configure -prefix=/home/me

Numpy install:
python setup.py config --compiler=intel --fcompiler=intel install --home=/home/me

Scipy install:
python
setup.py config --compiler=intel --fcompiler=intel build_ext
--fcompiler=intel -L/opt/intel/cce/10.1.008/lib  install --home=/home/me

Here's something that bothers me: when I start my Python interpreter which was compiled with icc, this is what I see:
Python 2.5.4 (r254:67916, Apr 22 2009, 15:52:10) 
[GCC 4.1.1 20070105 (Red Hat 4.1.1-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Why
does the interpreter think it's built with GCC?  Can that be causing
problems with SciPy?  How can I ensure that only Intel libraries are
used?

    Craig