[Numpy-discussion] performance matrix multiplication vs. matlab

Wed Jul 22 14:34:32 EDT 2009

Sorry.  I meant to update this thread after I had resolved my issue.
This was indeed one problem.  I had to set LD_LIBRARY_PATH.

I also had another odd problem that I will spell out here in hopes
that I save someone some trouble.  Specifically, one should be very
sure that the path to the blas that was compiled is correct when you
configure ATLAS because it does not indicate any problems if it is
not.   Specifically, I tried compiling blas with make -j3 to get all
my cores compiling at the same time but this actually caused a failure
that I did not notice.  It did create a temp_LINUX.a file in the right
place so I configured ATLAS against that.  Alas, many of the symbols
needed were not contained in this file as BLAS had failed to compile.
This was fairly hard to debug but once I got blas recompiled properly
without the -j 3 switch I was able to follow the rest of the steps and
everything works well.

Thanks,
Jonathan.

On Sun, Jul 19, 2009 at 11:35 PM, Nicolas Pinto<pinto at mit.edu> wrote:
> Jonathan,
>
> What does "ldd /home/jtaylor/lib/python2.5/site-packages/numpy/linalg/lapack_lite.so"
> say ?
>
> You need to make sure that it's using the libraries in /usr/local/lib.
> You can remove the ones in /usr/lib or "export
> LD_LIBRARY_PATH=/usr/local/lib/:$LD_LIBRARY_PATH".
>
> Hope it helps.
>
> Best,
>
> N
>
> On Fri, Jul 17, 2009 at 3:57 PM, Jonathan
> Taylor<jonathan.taylor at utoronto.ca> wrote:
>> Following these instructions I have the following problem when I
>> import numpy.  Does anyone know why this might be?
>>
>> Thanks,
>> Jonathan.
>>
>>>>> import numpy
>> Traceback (most recent call last):
>>  File "<stdin>", line 1, in <module>
>>  File "/home/jtaylor/lib/python2.5/site-packages/numpy/__init__.py",
>> line 130, in <module>
>>    import add_newdocs
>>  File "/home/jtaylor/lib/python2.5/site-packages/numpy/add_newdocs.py",
>> line 9, in <module>
>>    from lib import add_newdoc
>>  File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/__init__.py",
>> line 13, in <module>
>>    from polynomial import *
>>  File "/home/jtaylor/lib/python2.5/site-packages/numpy/lib/polynomial.py",
>> line 18, in <module>
>>    from numpy.linalg import eigvals, lstsq
>>  File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/__init__.py",
>> line 47, in <module>
>>    from linalg import *
>>  File "/home/jtaylor/lib/python2.5/site-packages/numpy/linalg/linalg.py",
>> line 22, in <module>
>>    from numpy.linalg import lapack_lite
>> ImportError: /usr/local/lib/libptcblas.so: undefined symbol: ATL_cpttrsm
>>
>>
>> On Sat, Jun 6, 2009 at 12:59 PM, Chris Colbert<sccolbert at gmail.com> wrote:
>>> since there is demand, and someone already emailed me, I'll put what I
>>> did in this post. It pretty much follows whats on the scipy website,
>>> with a couple other things I gleaned from reading the ATLAS install
>>> guide:
>>>
>>> and here it goes, this is valid for Ubuntu 9.04 64-bit  (# starts a
>>> comment when working in the terminal)
>>>
>>>
>>> download lapack 3.2.1 http://www.netlib.org/lapack/lapack.tgz
>>> download atlas 3.8.3
>>> http://sourceforge.net/project/downloading.php?group_id=23725&filename=atlas3.8.3.tar.bz2&a=65663372
>>>
>>> create folder  /home/your-user-name/build/atlas   #this is where we build
>>> create folder /home/your-user-name/build/lapack #atlas and lapack
>>>
>>> extract the folder lapack-3.2.1 to /home/your-user-name/build/lapack
>>> extract the contents of atlas to /home/your-user-name/build/atlas
>>>
>>>
>>>
>>> now in the terminal:
>>>
>>> # remove g77 and get stuff we need
>>> sudo apt-get remove g77
>>> sudo apt-get install gfortran
>>> sudo apt-get install build-essential
>>> sudo apt-get install python-dev
>>> sudo apt-get install python-setuptools
>>> sudo easy_install nose
>>>
>>>
>>> # build lapack
>>> cd /home/your-user-name/build/lapack/lapack-3.2.1
>>> cp INSTALL/make.inc.gfortran make.inc
>>>
>>> gedit make.inc
>>> #################
>>> #in the make.inc file make sure the line   OPTS = -O2 -fPIC -m64
>>> #and    NOOPTS = -O0 -fPIC -m64
>>> #the -m64 flags build 64-bit code, if you want 32-bit, simply leave
>>> #the -m64 flags out
>>> #################
>>>
>>> cd SRC
>>>
>>> #this should build lapack without error
>>> make
>>>
>>>
>>>
>>> # build atlas
>>>
>>> cd /home/your-user-name/build/atlas
>>>
>>> #this is simply where we will build the atlas
>>> #libs, you can name it what you want
>>> mkdir Linux_X64SSE2
>>>
>>> cd Linux_X64SSE2
>>>
>>> #need to turn off cpu-throttling
>>> sudo cpufreq-selector -g performance
>>>
>>> #if you don't want 64bit code remove the -b 64 flag. replace the
>>> #number 2400 with your CPU frequency in MHZ
>>> #i.e. my cpu is 2.53 GHZ so i put 2530
>>> ../configure -b 64 -D c -DPentiumCPS=2400 -Fa  -alg -fPIC
>>> --with-netlib-lapack=/home/your-user-name/build/lapack/lapack-3.2.1/Lapack_LINUX.a
>>>
>>> #the configure step takes a bit, and should end without errors
>>>
>>>  #this takes a long time, go get some coffee, it should end without error
>>> make build
>>>
>>> #this will verify the build, also long running
>>> make check
>>>
>>> #this will test the performance of your build and give you feedback on
>>> #it. your numbers should be close to the test numbers at the end
>>> make time
>>>
>>> cd lib
>>>
>>> #builds single threaded .so's
>>> make shared
>>>
>>> #builds multithreaded .so's
>>> make ptshared
>>>
>>> #copies all of the atlas libs (and the lapack lib built with atlas)
>>> #to our lib dir
>>> sudo  cp  *.so  /usr/local/lib/
>>>
>>>
>>>
>>> #now we need to get and build numpy
>>>
>>> download numpy 1.3.0
>>> http://sourceforge.net/project/downloading.php?group_id=1369&filename=numpy-1.3.0.tar.gz&a=93506515
>>>
>>> extract the folder numpy-1.3.0 to /home/your-user-name/build
>>>
>>> #in the terminal
>>>
>>> cd /home/your-user-name/build/numpy-1.3.0
>>> cp site.cfg.example site.cfg
>>>
>>> gedit site.cfg
>>> ###############################################
>>> # in site.cfg uncomment the following lines and make them look like these
>>> [DEFAULT]
>>> library_dirs = /usr/local/lib
>>> include_dirs = /usr/local/include
>>>
>>> [blas_opt]
>>> libraries = ptf77blas, ptcblas, atlas
>>>
>>> [lapack_opt]
>>> libraries = lapack, ptf77blas, ptcblas, atlas
>>> ###################################################
>>> #if you want single threaded libs, uncomment those lines instead
>>>
>>>
>>> #build numpy- should end without error
>>> python setup.py build
>>>
>>> #install numpy
>>> python setup.py install
>>>
>>> cd /home
>>>
>>> sudo ldconfig
>>>
>>> python
>>>>>import numpy
>>>>>numpy.test()   #this should run with no errors (skipped tests and known-fails are ok)
>>>>>a = numpy.random.randn(6000, 6000)
>>>>>numpy.dot(a, a)     # look at your cpu monitor and verify all cpu cores are at 100% if you built with threads
>>>
>>>
>>> Celebrate with a beer!
>>>
>>>
>>> Cheers!
>>>
>>> Chris
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Jun 6, 2009 at 10:42 AM, Keith Goodman<kwgoodman at gmail.com> wrote:
>>>> On Fri, Jun 5, 2009 at 2:37 PM, Chris Colbert <sccolbert at gmail.com> wrote:
>>>>> I'll caution anyone from using Atlas from the repos in Ubuntu 9.04  as the
>>>>> package is broken:
>>>>>
>>>>> https://bugs.launchpad.net/ubuntu/+source/atlas/+bug/363510
>>>>>
>>>>>
>>>>> just build Atlas yourself, you get better performance AND threading.
>>>>> Building it is not the nightmare it sounds like. I think i've done it a
>>>>> total of four times now, both 32-bit and 64-bit builds.
>>>>>
>>>>> If you need help with it,  just email me off list.
>>>>
>>>> That's a nice offer. I tried building ATLAS on Debian a year or two
>>>> ago and got stuck.
>>>>
>>>> Clear out your inbox!
>>>> _______________________________________________
>>>> Numpy-discussion mailing list
>>>> Numpy-discussion at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>>
>>> _______________________________________________
>>> Numpy-discussion mailing list
>>> Numpy-discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> Nicolas Pinto
> Ph.D. Candidate, Brain & Computer Sciences
> Massachusetts Institute of Technology, USA
> http://web.mit.edu/pinto
>