[Numpy-discussion] Help building numpy
Ian Goodfellow
goodfellow.ian at gmail.com
Fri Oct 8 13:14:11 EDT 2010
I don't think that the problem is my libraries. I think the problem is that
numpy is not using them. If I build numpy with "BLAS=None LAPACK=None
ATLAS=None sudo python setup.py install" my benchmark program takes 8
seconds, the same as if I build it with atlas supposedly included.
I built lapack and atlas using this configure command for atlas (the new
version of atlas will also build lapack for you):
../configure -b 64 -Fa alg -fPIC
--with-netlib-lapack-tarfile=/home/ia3n/Libraries/lapack-3.1.1.tgz
followed by make build, then in the directory the .a files, make shared,
make ptshared, then back in the main build directory, make check, make time,
sudo make install.
I think the libraries were built fine because my numbers from 'make time'
are mostly better than reference. The only one that is worse than reference
is BIG_MM (which is why I am testing numpy by multiplying two big matrices)
but there the difference is not that big. I get 733% of clock rate while the
reference authors got 786.1. This difference is not big enough to explain
numpy taking 8 seconds to multiply two 1k x 1k matrices. I think it is
probably explainable by me having a faster CPU (mine is 2794 Mhz, reference
was 2672), so memory-bound operations are not able to achieve quite as high
a percentage of the clock rate.
I don't understand what your timeit examples are doing. You get the same
number printed out whether you do the matrix multiplication or not. How is
it actually timing the matrix multiplication? In any case, if I run them on
my machine, here are the numbers I get:
supposedly with atlas:
doing matrix multiply: 0.0128 usec per loop
not doing matrix multiply: 0.0132
built using "BLAS=None LAPACK=None ATLAS=None":
doing matrix multiply: 0.0129 usec per loop
not doing matrix multiply: 0.0131
On Fri, Oct 8, 2010 at 12:35 PM, Bruce Southey <bsouthey at gmail.com> wrote:
> On 10/08/2010 10:58 AM, Ian Goodfellow wrote:
>
> Here's the output on my atlas library:
> file -L /usr/local/atlas/lib/libatlas.so
> /usr/local/atlas/lib/libatlas.so: ELF 64-bit LSB shared object, x86-64,
> version 1 (SYSV), dynamically linked, not stripped
>
> It looks mostly the same as yours except it says "not stripped" while yours
> says "stripped." Do you know what that means?
>
> Here's my timing code (but the difference between installations is pretty
> obvious even without a timer, 8 full seconds on my machine versus .2 seconds
> on a slower machine at work):
>
> import numpy as N
> import time
> rng = N.random.RandomState([0,1,2])
> A = rng.randn(1000,1000)
> t1 = time.time(); x = N.dot(A,A); t2 = time.time()
> print t2-t1
>
>
>
>
> On Fri, Oct 8, 2010 at 11:49 AM, Bruce Southey <bsouthey at gmail.com> wrote:
>
>> On 10/08/2010 10:01 AM, Ian Goodfellow wrote:
>>
>> I'm using 64-bit Ubuntu 10.04. I originally tried building without
>> site.cfg and got the same result. After that I removed the installation and
>> the numpy/build directory and tried again with site.cfg.
>>
>> Here's the otuput of my show_config(). Does this mean it's actually using
>> atlas? I ran 'make time' in the atlas directory and got higher % of clock
>> speed numbers than the reference so I would be surprised if my atlas build
>> is this slow.
>>
>> atlas_threads_info:
>> libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
>> library_dirs = ['/usr/local/atlas/lib']
>> define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>> language = f77
>> include_dirs = ['/usr/include']
>> blas_opt_info:
>> libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
>> library_dirs = ['/usr/local/atlas/lib']
>> define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>> language = c
>> include_dirs = ['/usr/include']
>> atlas_blas_threads_info:
>> libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
>> library_dirs = ['/usr/local/atlas/lib']
>> define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>> language = c
>> include_dirs = ['/usr/include']
>> lapack_opt_info:
>> libraries = ['lapack', 'lapack', 'f77blas', 'cblas', 'atlas']
>> library_dirs = ['/usr/local/atlas/lib']
>> define_macros = [('ATLAS_INFO', '"\\"None\\""')]
>> language = f77
>> include_dirs = ['/usr/include']
>> lapack_mkl_info:
>> NOT AVAILABLE
>> blas_mkl_info:
>> NOT AVAILABLE
>> mkl_info:
>> NOT AVAILABLE
>>
>>
>> On Fri, Oct 8, 2010 at 10:53 AM, Bruce Southey <bsouthey at gmail.com>wrote:
>>
>>> On 10/08/2010 09:06 AM, Benjamin Root wrote:
>>>
>>> On Fri, Oct 8, 2010 at 8:47 AM, Ian Goodfellow <
>>> goodfellow.ian at gmail.com> wrote:
>>>
>>>> Can anyone explain how to get numpy to recognize atlas? I have atlas
>>>> built and installed, and I put what I thought should be sufficient, based on
>>>> numpy's INSTALL.TXT, to make numpy use atlas in my site.cfg then ran python
>>>> setup.py install. The resulting build is incredibly slow. Multiplying a
>>>> 1,000 x 1,000 matrix takes 8 seconds. The default packages take .8 . On a
>>>> computer at work with a similar but slower cpu to mine, it only takes .2, so
>>>> I know I should be able to get a lot more speed out of numpy.
>>>>
>>>> This is my site.cfg:
>>>>
>>>> [atlas]
>>>> atlas_libs = lapack, f77blas, cblas, atlas
>>>>
>>>> [DEFAULT]
>>>> library_dirs = /usr/local/atlas/lib
>>>> include_dir = /usr/local/atlas/include
>>>>
>>>> Thanks in advance,
>>>> Ian
>>>>
>>>>
>>> Ian, it would be very helpful if you could post the output from the build
>>> of numpy. That log can tell us whether it found the libraries or not when
>>> building.
>>>
>>> Ben Root
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>> I do not use a 'site.cfg' even though I build my own numpy but use the
>>> standard Fedora (13) packages for atlas. So try it without defining it -
>>> remove the installation and build directories first!
>>>
>>> Can you please provide the operating system (including 32bit or 64 bit),
>>> Python version and numpy version?
>>>
>>> What is the output of the numpy function 'show_config()'?
>>> I added mine below from a 64-bit linux system so you can see that numpy
>>> is looking in the '/usr/lib64' directory.
>>>
>>> Bruce
>>>
>>> >>> import numpy as np
>>> >>> np.show_config()
>>> atlas_threads_info:
>>> libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>> library_dirs = ['/usr/lib64/atlas']
>>> define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>> language = f77
>>> include_dirs = ['/usr/include']
>>> blas_opt_info:
>>> libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>> library_dirs = ['/usr/lib64/atlas']
>>> define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>> language = c
>>> include_dirs = ['/usr/include']
>>> atlas_blas_threads_info:
>>> libraries = ['ptf77blas', 'ptcblas', 'atlas']
>>> library_dirs = ['/usr/lib64/atlas']
>>> define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>> language = c
>>> include_dirs = ['/usr/include']
>>> lapack_opt_info:
>>> libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
>>> library_dirs = ['/usr/lib64/atlas']
>>> define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
>>> language = f77
>>> include_dirs = ['/usr/include']
>>> lapack_mkl_info:
>>> NOT AVAILABLE
>>> blas_mkl_info:
>>> NOT AVAILABLE
>>> mkl_info:
>>> NOT AVAILABLE
>>> >>>
>>>
>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing listNumPy-Discussion at scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>> It appears to finding something but I do not know enough about that
>> aspect.
>>
>> I presume that you installed your own version of atlas since you are using
>> /usr/local. So you need to be careful of the flags used especially that it
>> is 64-bit and uses the appropriate SSE instructions for your processor. For
>> example, on my system:
>>
>> $ file -L /usr/lib64/atlas/libatlas.so
>> /usr/lib64/atlas/libatlas.so: ELF 64-bit LSB shared object, x86-64,
>> version 1 (SYSV), dynamically linked, stripped
>>
>> Can you please provide the code you are using for timing for the 'slow'
>> system?
>>
>> By the way, there was this thread:
>> 'HOWTO build NumPy without external math libs'
>> http://mail.scipy.org/pipermail/numpy-discussion/2010-July/051513.html
>>
>> BLAS=None LAPACK=None ATLAS=None python setup.py build
>>
>>
>> Bruce
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> _______________________________________________
> NumPy-Discussion mailing listNumPy-Discussion at scipy.orghttp://mail.scipy.org/mailman/listinfo/numpy-discussion
>
> Don't use time!
> Use timeit.
>
> I suspect your build is using numerical libraries that have not been build
> correctly for your system. But I don't know anything about that.
>
> What are the flags used to build lapack and atlas?
>
> I told the lazy way of using timeit but may help you identify the slowness.
> To capture the output I use two files for stdout and stderr:
>
> $ python setup.py build > b.log 2>e.log
> $ python setup.py install # done as root
>
> 1) Using numpy built as:
> 'BLAS=None LAPACK=None ATLAS=None python setup.py build'
>
> $ python -m timeit -s "import numpy as N; rng =
> N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000); x = N.dot(A,A)"
> 100000000 loops, best of 3: 0.012 usec per loop
>
> 2) Using numpy built just without Atlas:
> 'ATLAS=None python setup.py build'
> $ python -m timeit -s "import numpy as N; rng =
> N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000); x = N.dot(A,A)"
> 100000000 loops, best of 3: 0.012 usec per loop
>
> 3) Using Atlas without dot:
> $ python -m timeit -s "import numpy as N; rng =
> N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000)"
> 100000000 loops, best of 3: 0.012 usec per loop
>
> 4) Using Atlas with dot:
> $ python -m timeit -s "import numpy as N; rng =
> N.random.RandomState([0,1,2]) ; A = rng.randn(1000,1000); x = N.dot(A,A)"
> 100000000 loops, best of 3: 0.0131 usec per loop
>
>
>
> Bruce
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20101008/e6ddf544/attachment.html>
More information about the NumPy-Discussion
mailing list