[Numpy-discussion] numpy with icc
rex
rex at nosyntax.com
Sun May 18 06:13:38 EDT 2008
> I am trying to build numpy with intel icc and mkl. I don't understand
> a lot of what I am doing.
Me, too. I have built it with icc & MKL several times in the past,
but cannot build the numpy svn with MKL now. I can build it with
icc and no MKL, and it passes all the tests with no errors.
I've interleaved my setup with yours to make it easy to compare them.
> ----------------------------------------------------------------------------------
> site.cfg:
> [DEFAULT]
> library_dirs = /opt/intel/mkl/10.0.3.020/lib/em64t,/usr/lib64,/usr/local/lib64,/usr/local/python2.5.2-intel/lib,/usr/lib,/usr/local/lib
> include_dirs = /opt/intel/mkl/10.0.3.020/include,/usr/include,/usr/local/include,/usr/local/python2.5.2-intel/include
> [mkl]
> include_dirs = /opt/intel/mkl/10.0.3.020/include
> library_dirs = /opt/intel/mkl/10.0.3.020/lib/em64t
> lapack_libs = mkl_lapack
> [lapack_src]
> libraries=mkl_lapack,mkl,guide
> [lapack_info]
> libraries=mkl_lapack,mkl,guide
> -------------------------------------------------------------------------------
[DEFAULT]
library_dirs = /opt/intel/mkl/10.0.3.020/lib/32
include_dirs = /opt/intel/mkl/10.0.3.020/10.0.3.020/include
[mkl]
library_dirs = /opt/intel/mkl/10.0.3.020/lib/32
lapack_libs = mkl, mkl_lapack, guide
#my reading of the Intel MKL docs suggests that vml does not need
to be explicitly added, but that mkl_lapack does.
> intelccompiler.py:
> from distutils.unixccompiler import UnixCCompiler
> from numpy.distutils.exec_command import find_executable
> class IntelCCompiler(UnixCCompiler):
> """ A modified Intel compiler compatible with an gcc built Python.
> """
> compiler_type = 'intel'
> cc_exe = 'icc -g -O3 -w -fPIC -parallel -ipo -xT -axT'
cc_exe = 'icc -msse3 -fast' #adjust to suit your cpu
I think some of those flags can be omitted: drop -g, and -fast
implies several other flags, e.g., -xT, IIRC. (I'm building on a
Core 2 Duo)
> def __init__ (self, verbose=0, dry_run=0, force=0):
> UnixCCompiler.__init__ (self, verbose,dry_run, force)
> compiler = self.cc_exe
> self.set_executables(compiler=compiler,
> compiler_so=compiler,
> compiler_cxx=compiler,
> linker_exe=compiler,
> linker_so=compiler + ' -shared')
> class IntelItaniumCCompiler(IntelCCompiler):
> compiler_type = 'intele'
> # On Itanium, the Intel Compiler used to be called ecc, let's search for
> # it (now it's also icc, so ecc is last in the search).
> for cc_exe in map(find_executable,['icc','ecc']):
> if cc_exe:
> break
> ----------------------------------------------------------------
> system_info.py:
> ....
> class lapack_mkl_info(mkl_info):
> def calc_info(self):
> mkl = get_info('mkl')
> if not mkl:
> return
> if sys.platform == 'win32':
> lapack_libs = self.get_libs('lapack_libs',['mkl_lapack'])
> else:
> lapack_libs = self.get_libs('lapack_libs',['mkl_lapack'])
> info = {'libraries': lapack_libs}
> dict_append(info,**mkl)
> self.set_info(**info)
> ...
> ------------------------------------------------------------------------
I made the same change in the above. It was originally:
if sys.platform == 'win32':
lapack_libs = self.get_libs('lapack_libs',['mkl_lapack'])
else:
lapack_libs = self.get_libs('lapack_libs',['mkl_lapack32','mkl_lapack64]')
Which does not work because all 3 libs are now named mkl_lapack.
I made another change in system.info.py:
class mkl_info(system_info):
section = 'mkl'
dir_env_var = 'MKL'
_lib_mkl = ['mkl','mkl_lapack','guide']
It originally had 'vml' instead of 'mkl_lapack'
> i then use this command to compile:
> /usr/local/python2.5.2-intel/bin/python setup.py config
> --compiler=intel config_fc --fcompiler=intel \
> --opt='-fPIC -O3 -w -axT -xT' install > build.out
I used:
numpy# python setup.py config --compiler=intel build_clib \
--compiler=intel build_ext --compiler=intel install \
--prefix=/usr/local> build51
> build.out has this in it:
> F2PY Version 2_4422
> blas_opt_info:
> blas_mkl_info:
> libraries mkl,vml,guide not found in /opt/intel/mkl/10.0.3.020/lib/em64t
> NOT AVAILABLE
> atlas_blas_threads_info:
> Setting PTATLAS=ATLAS
> NOT AVAILABLE
> atlas_blas_info:
> NOT AVAILABLE
> blas_info:
> NOT AVAILABLE
> blas_src_info:
> NOT AVAILABLE
> NOT AVAILABLE
> lapack_opt_info:
> lapack_mkl_info:
> mkl_info:
> libraries mkl,vml,guide not found in /opt/intel/mkl/10.0.3.020/lib/em64t
> NOT AVAILABLE
> NOT AVAILABLE
> atlas_threads_info:
> Setting PTATLAS=ATLAS
> numpy.distutils.system_info.atlas_threads_info
> NOT AVAILABLE
> atlas_info:
> numpy.distutils.system_info.atlas_info
> NOT AVAILABLE
> lapack_info:
> NOT AVAILABLE
> lapack_src_info:
> NOT AVAILABLE
> NOT AVAILABLE
build53 (I used svn 5183) has:
F2PY Version 2_5183
blas_opt_info:
blas_mkl_info:
FOUND:
libraries = ['mkl', 'mkl_lapack', 'guide', 'pthread']
library_dirs = ['/opt/intel/mkl/10.0.3.020/lib/32']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = []
FOUND:
libraries = ['mkl', 'mkl_lapack', 'guide', 'pthread']
library_dirs = ['/opt/intel/mkl/10.0.3.020/lib/32']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = []
lapack_opt_info:
lapack_mkl_info:
mkl_info:
FOUND:
libraries = ['mkl', 'mkl_lapack', 'guide', 'pthread']
library_dirs = ['/opt/intel/mkl/10.0.3.020/lib/32']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = []
FOUND:
libraries = ['mkl', 'mkl_lapack', 'guide', 'mkl', 'mkl_lapack', 'guide', 'pthread']
library_dirs = ['/opt/intel/mkl/10.0.3.020/lib/32']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = []
FOUND:
libraries = ['mkl', 'mkl_lapack', 'guide', 'mkl', 'mkl_lapack', 'guide', 'pthread']
library_dirs = ['/opt/intel/mkl/10.0.3.020/lib/32']
running config
running build_clib
[...]
> running config
> running config_fc
> unifing config_fc, config, build_clib, build_ext, build commands
> --fcompiler options
> running install
> running build
> running config_cc
> unifing config_cc, config, build_clib, build_ext, build commands
> --compiler options
> running build_src
> building py_modules sources
> creating build
> creating build/src.linux-x86_64-2.5
> creating build/src.linux-x86_64-2.5/numpy
> creating build/src.linux-x86_64-2.5/numpy/distutils
> building extension "numpy.core.multiarray" sources
> creating build/src.linux-x86_64-2.5/numpy/core
> Generating build/src.linux-x86_64-2.5/numpy/core/config.h
> Found executable /opt/intel/cce/10.1.015/bin/icc
> Could not locate executable ecc
> customize IntelFCompiler
> Found executable /opt/intel/fce/10.1.015/bin/ifort
> C compiler: icc -g -O3 -w -fPIC -parallel -ipo -xT -axT
> ......
> it seems to compile and install fine.... then i start python and try
> to run numpy.test(). that is where i am stuck. this is what happens:
> # pwd
> /usr/local/python2.5.2-intel/bin
> # ./python
> Python 2.5.2 (r252:60911, May 13 2008, 11:22:16)
> [GCC Intel(R) C++ gcc 4.1 mode] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import numpy
> >>> numpy.test()
> Numpy is installed in
> /usr/local/python2.5.2-intel/lib/python2.5/site-packages/numpy
> Numpy version 1.0.4
> Python version 2.5.2 (r252:60911, May 13 2008, 11:22:16) [GCC Intel(R)
> C++ gcc 4.1 mode]
> Found 10/10 tests for numpy.core.defmatrix
> Found 36/36 tests for numpy.core.ma
> Found 223/223 tests for numpy.core.multiarray
> Found 65/65 tests for numpy.core.numeric
> Found 31/31 tests for numpy.core.numerictypes
> Found 12/12 tests for numpy.core.records
> Found 6/6 tests for numpy.core.scalarmath
> Found 14/14 tests for numpy.core.umath
> Found 4/4 tests for numpy.ctypeslib
> Found 5/5 tests for numpy.distutils.misc_util
> Found 1/1 tests for numpy.fft.fftpack
> Found 3/3 tests for numpy.fft.helper
> Found 9/9 tests for numpy.lib.arraysetops
> Found 46/46 tests for numpy.lib.function_base
> Found 5/5 tests for numpy.lib.getlimits
> Found 4/4 tests for numpy.lib.index_tricks
> Found 3/3 tests for numpy.lib.polynomial
> Found 49/49 tests for numpy.lib.shape_base
> Found 15/15 tests for numpy.lib.twodim_base
> Found 43/43 tests for numpy.lib.type_check
> Found 1/1 tests for numpy.lib.ufunclike
> Found 40/40 tests for numpy.linalg
> Found 2/2 tests for numpy.random
> Found 0/0 tests for __main__
> MKL FATAL ERROR: /opt/intel/mkl/10.0.3.020/lib/em64t/: cannot read
> file data: Is a directory
> #
I get the same error:
python
Python 2.5 (release25-maint, Dec 9 2006, 14:35:53)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-20)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.test()
Numpy is installed in /usr/local/lib/python2.5/site-packages/numpy
Numpy version 1.1.0.dev5183
Python version 2.5 (release25-maint, Dec 9 2006, 14:35:53) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-20)]
Found 18/18 tests for numpy.core.defmatrix
Found 3/3 tests for numpy.core.memmap
Found 283/283 tests for numpy.core.multiarray
Found 70/70 tests for numpy.core.numeric
Found 36/36 tests for numpy.core.numerictypes
Found 12/12 tests for numpy.core.records
Found 7/7 tests for numpy.core.scalarmath
Found 16/16 tests for numpy.core.umath
Found 5/5 tests for numpy.ctypeslib
Found 5/5 tests for numpy.distutils.misc_util
Found 2/2 tests for numpy.fft.fftpack
Found 3/3 tests for numpy.fft.helper
Found 24/24 tests for numpy.lib._datasource
Found 10/10 tests for numpy.lib.arraysetops
Found 1/1 tests for numpy.lib.financial
Found 0/0 tests for numpy.lib.format
Found 53/53 tests for numpy.lib.function_base
Found 5/5 tests for numpy.lib.getlimits
Found 6/6 tests for numpy.lib.index_tricks
Found 15/15 tests for numpy.lib.io
Found 1/1 tests for numpy.lib.machar
Found 4/4 tests for numpy.lib.polynomial
Found 49/49 tests for numpy.lib.shape_base
Found 15/15 tests for numpy.lib.twodim_base
Found 43/43 tests for numpy.lib.type_check
Found 1/1 tests for numpy.lib.ufunclike
Found 89/89 tests for numpy.linalg
Found 94/94 tests for numpy.ma.core
Found 15/15 tests for numpy.ma.extras
Found 7/7 tests for numpy.random
Found 16/16 tests for numpy.testing.utils
Found 0/0 tests for __main__
MKL FATAL ERROR: /opt/intel/mkl/10.0.3.020/lib/32/: cannot read file data: Is a directory
> not sure how to get past this... although, I am sure, if numpy is not
> working right then i will not be able to go on and compile scipy.....
I don't understand how MKL got linked into your system. Whenever I
get no messages of the type:
FOUND:
libraries = ['mkl', 'mkl_lapack', 'guide', 'pthread']
library_dirs = ['/opt/intel/mkl/10.0.3.020/lib/32']
define_macros = [('SCIPY_MKL_H', None)]
include_dirs = []
in the build log, MKL doesn't get linked. If it IS linked it will
show up here:
# ldd /usr/local/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so
linux-gate.so.1 => (0xffffe000)
/opt/intel/mkl/10.0.3.020/lib/32/libmkl_intel.so (0xb7e62000)
/opt/intel/mkl/10.0.3.020/lib/32/libmkl_intel_thread.so (0xb7c65000)
/opt/intel/mkl/10.0.3.020/lib/32/libmkl_core.so (0xb7c01000)
libmkl_lapack.so => /opt/intel/mkl/10.0.3.020/lib/32/libmkl_lapack.so (0xb7736000)
libguide.so => /opt/intel/cc/10.1.015/lib/libguide.so (0xb76d4000)
libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb76a8000)
libimf.so => /opt/intel/cc/10.1.015/lib/libimf.so (0xb7478000)
libsvml.so => /opt/intel/cc/10.1.015/lib/libsvml.so (0xb73a7000)
libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7382000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7377000)
libintlc.so.5 => /opt/intel/cc/10.1.015/lib/libintlc.so.5 (0xb7334000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7202000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb71fe000)
/lib/ld-linux.so.2 (0x80000000)
More information about the NumPy-Discussion
mailing list