[Numpy-discussion] building NumPy with Intel CC & MKL
Christian Marquardt
christian at marquardt.sc
Wed Jan 24 13:46:10 EST 2007
Dear rex,
I'll try to explain... I hope it's not too basic.
Python is searching for its modules along the PYTHONPATH, i.e. a list
of directories where it expects to find whatever it needs. This is the
same as the Unix shell (or the DOC command.com) is looking in the PATH in
order
to find programs or shell /batch scripts, or the dynamic loader is using
LD_LIBRARY_PATH for finding shared libraries.
>> >>>> import numpy
>> >>>> print numpy
>> > <module 'numpy' from
>> '/usr/lib/python2.5/site-packages/numpy/__init__.pyc'>
>> >
>> > What am I to make of this? Is it the rpm numpy or is it the numpy I
>> > built using the Intel compiler and MKL?
This tells from which directory your Python installation actually loaded
numpy from: It used the numpy installed in the directory
/usr/lib/python2.5/site-packages/numpy
By *convention* (as someone already pointed out before), the
/usr/lib/python2.5/site-packages is the directory where the original
system versions of python packages should be installed. In particular, the
rpm version will very likely install it's stuff there.
When installing additional python modules or packages via a command like
python setup.py install
the new packages will also be installed in that system directory. So if
you have installed your Intel version of numpy with the above command, you
might have overwritten the rpm stuff. There is a way to install in a
different place; more on that below.
You now probably want to find out if the numpy version in /usr/lib/... is
the Intel one or the original rpm one. To do this, you can check if the
MKL and Intel libraries are actually loaded by the shared libraries within
the numpy installation. You can use the command ldd which shows which
shared libraries are loaded by executables or other shared libraries. For
example, in my installation, the command
ldd <wherever>/python2.5/site-packages/numpy/linalg/lapack_lite.so
gives the following output:
MEDEA /opt/apps/lib/python2.5/site-packages/numpy>ldd
./linalg/lapack_lite.so
linux-gate.so.1 => (0xffffe000)
libmkl_lapack32.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack32.so
(0x40124000)
libmkl_lapack64.so => /opt/intel/mkl/8.1/lib/32/libmkl_lapack64.so
(0x403c8000)
libmkl.so => /opt/intel/mkl/8.1/lib/32/libmkl.so (0x40692000)
libvml.so => /opt/intel/mkl/8.1/lib/32/libvml.so (0x406f3000)
libguide.so => /opt/intel/mkl/8.1/lib/32/libguide.so (0x4072c000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x40785000)
libimf.so => /opt/intel/fc/9.1/lib/libimf.so (0x40797000)
libm.so.6 => /lib/tls/libm.so.6 (0x409d5000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x409f8000)
libirc.so => /opt/intel/fc/9.1/lib/libirc.so (0x40a00000)
libc.so.6 => /lib/tls/libc.so.6 (0x40a41000)
libdl.so.2 => /lib/libdl.so.2 (0x40b5b000)
/lib/ld-linux.so.2 (0x80000000)
Note that the MKL libraries are referenced at the beginning - just look at
the path names! If the output for your lapack_lite.so also contains
references to the MKL libs, you've got the Intel version in
/usr/lib/python2.5/.... (and have probably overwritten the rpm version).
If you do not get any reference to the MKL stuff, it's still the rpm
version which does not use the MKL.
Now, let's assume that you have the rpm version in /usr/lib/python2.5/....
Maybe you'll want to reinstall the rpm to be sure that this is the case.
You now want to a) install your Intel version in some well-defined place,
and b) make sure that your Python picks that version up when importing
numpy.
To achieve a) one way is to reinstall numpy from the source as before, BUT
with
python setup.py --prefix=<somewhere>
^^^^^^^^^^^^^^^^^^^^^
<somewhere> is the path to some directory, e.g.
python setup.py install --prefix=$HOME
The latter would install numpy into the directory
$HOME/lib/python2.5/site-packages/numpy
Do an ls afterwards to check if numpy really arrived there. Instead of
using the environment variable HOME, you can of course also any other
directory you like. I'll stick to HOME in the following.
For b), we have to tell python that modules are waiting for it to be
picked up in $HOME/lib/python2.5/site-packages. You do that by setting the
environment variable PYTHONPATH, as was also mentioned in this thread. In
our example, you would do (for a bash or ksh)
export PYTHONPATH=$HOME/lib/python2.5/site-packages
As long as this variable is set and exported (i.e., visible in the
environment of every program you start), the next instance of Python
you'll start will now begin searching for modules in PYTHONPATH whenever
you do an import, and only fall back to the ones in the system wide
installation if it doesn't find the required module in PYTHONPATH.
So, after having set PYTHONPATH in your environment, start up python and
import numpy. Do the 'print numpy' within python again and look at the
output. Does it point to the installation directory of your Intel version?
Great; you're done. If not, this means that something went wrong. It might
be that you had a typo in the export command or the directory name; it
might mean that you didn't export the PYTHONPATH before running python; it
might be that the installation had failed for some reason. You just have
to play around a bit and see what's going on... but it's not difficult.
Now that you have two versions of numpy, you can (kind of) switch between
them by making use of the PYTHONPATH. If you unset it ('unset
PYTHONPATH'), the next python session you are starting in the same
shell/window will use the original system version. Setting PYTHONPATH
again and having it point to your local site-packages directory activates
the stuff you've installed in there. You cannot switch between the two
numpy versions in the same session); if you want to try the other, you'll
have to start a new python and make sure that the PYTHONPATH is set up
appropriately for what you want.
In the long run, and if you have decided which version to use, you can
export PYTHONPATH in your $HOME/.profile and don't have to do that
manually each time (which becomes quite cumbersome after a while, of
course).
Common practice is probably that you install your favourite versions or
builds of python modules in one place (i.e. using $HOME as --prefix), and
set PYTHONPATH accordingly. It's not a good idea to overwrite the system
wide installations, but again - that's purely a convention, nothing more.
Hope this helps a bit... Good luck!
Christian.
>>
>> Did you change the distutils installation location? See this page for
>> the
>> various ways to do that:
>>
>> http://docs.python.org/inst/alt-install-windows.html
>>
>> If not, then I strongly suspect that you overwrote the numpy that you
>> installed
>> by RPM.
>
> Thanks for the URL. It would not have occurred to me to read
> "alt-install-windows" since I don't do Windows, and haven't since 1999.
>
> I did not change the distutils installation location.
>
> How do I determine what is loaded when "import numpy" is issued?
> That is, how does one distinguish between the RPM and the NumPy built
> using the Intel compiler and MKL? I suppose I can build NumPy under my
> user directory and reinstall the RPM, and then run some benchmarks.
>
> This is exceedingly frustrating; it's not new -- numpy/scipy are very
> important to me, but they have been very difficult to install under SUSE
> for years.
>
> I'm very grateful for the selfless contributions of those who have
> created numpy/scipy, but if I -- a highest honors grad -- have great
> difficulty making it work, what about others?
>
> -rex
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
More information about the NumPy-Discussion
mailing list