[SciPy-user] some benchmark data for numarray, Numeric and scipy-newcore

[Gerard Vermeulen]

I have benchmarked some array manipulations using the default
Numeric and numarray on my Mandrake-10.2 and a recent snapshot
of the new scipy core.

I used the following script which takes two arguments: a string
indicating which Numerical extension to use and an integer
setting the size of the arrays.

-- start script --
#!/usr/bin/env python

import sys
import time

def klein(nu, nv, a):
    """Returns the figure-8 form of the Klein bottle
    u in [0, 2pi), v in [0, 2pi), and a > 2
    http://mathworld.wolfram.com/KleinBottle.html    
    """
    assert(nu > 2)
    assert(nv > 2)
    assert(a > 2)
    ticks = [time.time()]                                    # label
    i = arange(nu*nv)                                        #     1
    ticks.append(time.time())
    u = i % nu                                               #     2
    ticks.append(time.time())
    u %= nu-1                                                #     3
    ticks.append(time.time())
    u = 2*pi*u/(nu-1)                                        #     4
    ticks.append(time.time())
    v = i / nu                                               #     5
    ticks.append(time.time())
    v %= nv-1                                                #     6
    ticks.append(time.time())
    v = 2*pi*v/(nu-1)                                        #     7
    ticks.append(time.time())
    xyzs = zeros((nu*nv, 3), Float)                          #     8
    ticks.append(time.time()) 
    xyzs[:,0] = (a+cos(u/2)*sin(v)-sin(u/2)*sin(2*v))*cos(u) #     9
    ticks.append(time.time())
    xyzs[:,1] = (a+cos(u/2)*sin(v)-sin(u/2)*sin(2*v))*sin(u) #    10
    ticks.append(time.time())
    xyzs[:,2] = sin(u/2)*sin(v)+cos(u/2)*sin(2*v)            #    11
    ticks.append(time.time())
    print 'label    time (s)'
    for i in range(1, len(ticks)):
        print '%5d    %s' %(i, ticks[i] - ticks[i-1])
    print 'TOTAL     %s' % (ticks[-1] - ticks[0])
    return xyzs

# klein()

def usage():
    print 'Usage: python numpy size'
    print 'where numpy must be Numeric, numarray, or scipy'
    print 'and size an integer in [3, 13)'
    sys.exit(1)

# usage()

if __name__ == '__main__':
    if len(sys.argv) != 3:
        usage()

    if sys.argv[1] == 'Numeric':
        from Numeric import *
        import Numeric
        extension = 'Numeric-%s' % Numeric.__version__
    elif sys.argv[1] == 'numarray':
        from numarray import *
        import numarray
        extension = 'numarray-%s' % numarray.__version__
    elif sys.argv[1] == 'scipy':
        from scipy import *
        extension = 'scipy-%s' % core_version.version
    else:
        usage()

    # on my system:
    # numarray is slowest when size = 6
    # numarray is fastest when size = 7
    # memory errors occur when size = 13
    try:
        size = int(sys.argv[2])
    except:
        usage()
    if size < 3 or size >= 13:
        usage()
        
    print '%s: benchmark size = %s' % (extension, size)
    klein(2**size, 2**size, pi)

# Local Variables: ***
# mode: python ***
# End: ***
-- end script --

The results follow below, where the label column indicates the corresponding
statement in the klein() function above:

[packer at titan JUNK]$ ./bench.py Numeric 12
Numeric-23.1: benchmark size = 12
label    time (s)
    1    0.435407161713
    2    0.267067909241
    3    0.192448139191
    4    1.01883888245
    5    0.273994922638
    6    0.202347040176
    7    1.01867508888
    8    0.650615930557
    9    10.3847429752
   10    10.4006090164
   11    8.59523797035
TOTAL     33.4399850368
[packer at titan JUNK]$ ./bench.py scipy 12
Importing test to scipy
Importing base to scipy
Importing basic to scipy
scipy-0.7.1.1526: benchmark size = 12
label    time (s)
    1    0.406187057495
    2    0.515972852707
    3    0.45333313942
    4    2.09451985359
    5    0.277112007141
    6    0.50949215889
    7    2.08596587181
    8    0.456773996353
    9    8.81630802155
   10    8.83214116096
   11    7.41638493538
TOTAL     31.8641910553
[packer at titan JUNK]$ ./bench.py numarray 12
numarray-1.2.3: benchmark size = 12
label    time (s)
    1    0.0770111083984
    2    0.245344877243
    3    0.186748027802
    4    0.553218126297
    5    0.245689868927
    6    0.186288118362
    7    0.626587867737
    8    0.456372022629
    9    7.61383700371
   10    8.03238511086
   11    6.3676469326
TOTAL     24.5911290646
[packer at titan JUNK]$

Conclusion:
- the overal performance of numarray is 23 % better than scipy-newcore and
  27 % better than Numeric.
- numarray is consistently faster than the other packages.
- scipy newcore is on average somewhat faster than Numeric3, but some operations
  are really slow in comparison with the other packages. In partical the
  statements labeled 2, 3, 4, 6 and 7 take 2 times more time using scipy-newcore
  than using Numeric.

Gerard