[SciPy-dev] Thoughts on weave improvements

[Pat Miller]

Prabhu Ramachandran wrote:

>>>>>>"PM" == Pat Miller <pnmiller at pacbell.net> writes:
>     PM> bytecode compiler work I started under PyCOD (compile on
>     PM> demand) that builds C++ accelerated extension functions from
>     PM> Python

> All this sounds very promising.  I just hope it becomes part of weave


Guess what.... It works (some restrictions and caveats of course :-)

Here's how I have it set up

import weaver # This is my prototype
import weave.ext_tools  # These are Eric's
import weave.build_tools
from types import *  # Will need this in 2.1, but in 2.2 things change a bit

# Get one of those nice weave module builders
M = weave.ext_tools.ext_module('foobar')

# Define some handy python function
def f(x):
   "doc strings get propagated"
   return x*x

# Here's where we define the accelerated function
# Note that you need to give the input signature
# of the function
E.add_function( weaver.Python2CXX( f, [FloatType] ) )

# You can do it for multiple signatures, but you
# need to specify a different name to disambiguate
E.add_function( weaver.Python2CXX( f, [IntType], name = 'f_int' ) )

# OK... So accelerating that wasn't very impressive...
# You can do a lot of calculating stuff though....
from math import sin,cos,pi
def trigThing(x):
    "doc"
    # Local temporaries work
    xx = x*x

    # Some math functions work
    sinx = sin(x)  # Math functions of one arg are available

    # global variables are "read-only-frozen"
    PI2 = 2.0*pi
    cosx = cos(x+PI2) # Note the use of a global here... It's
                     # value gets frozen at compile time
                     # Not a problem for pi, but worrysome
                     # for other globals

    # 'if' works most of the time (integer test or boolean expr only!)
    if cosx < 0.0:
         cosx = abs(cosx) # In my prototype, only abs( float ) is on
    else:
         # Print works using the real Python sys.stdout, so weird
         # windows things will work as output goes where it is supposed to
         print "It was non-negative!",cosx


    return sinx*sinx + cosx*cosx

E.add_function( weaver.Python2CXX( trigThing, [FloatType] ) )

# Of course multiple input arguments...
def dist(x0,y0,z0, x1,y1,z1):
     deltax = x1-x0
     deltay = y1-y0
     deltaz = z1-z0
     return sqrt(deltax*deltax + deltay*deltay + deltaz*deltaz)

E.add_function ( weaver.Python2CXX( dist, 
[FloatType,FloatType,FloatType,FloatType,FloatType,FloatType,]) )

# Now the magic...
E.generate_file()
weave.build_tools.build_extension('foobar.cpp')
import foobar

# This prints 100.0
print foobar.f(10.0)

# OK... It isn't everything, but it can make a lot of simple stuff run
# about 2 to 5 times faster (real speed mostly eaten up with function call
# overhead for these really small functions)

Here are things that don't work...

* you can't use import
* you can't use loops of any kind
* no support for exceptions at this time (though doable!)
* no mixed mode arithmetic (x + 1 where x is a float dies!)
* only involves integer, float, string types
* no string operations except print and return
* specifically, no python formating ( ">>> %s <<<"%x is bad )
* not all operators are there (mostly laziness in prototype)
      ( +,-,*,/, <, <=, ==, !=, >, >= only for now)
* only single valued functions for the moment
* I don't use Eric's cool weave code caching scheme
* no index access
* etc...

Still, not too shabby for a day and a half development time.
I'll be patching up holes and writing some more extensive tests
for this.  There are three main enhancements that will get this
kicking butt...

0) Add global update (push back to python environment)
1) Add Numeric (I have an extensible interface for users adding new types)
2) Add __getitem__ and __setitem__ so that you could do something like...

def f(a,b):
     n = len(a)
     m = len(b)
     for i in range(n):
         for j in range(m):
             a[i] = b[j]
     return
(i.e. write normal F77 style scalarized code using numpy arrays)

3) Allow the functions you write to be 'inlined' automagically
e.g.

def f(x):
    some code

def g(x):
     .... + f(x) + ....

So that when you define f, [FloatType] as a function, then you can
use it immediately as part of g when you compile it

4) Build an interface to the struct module so that you can build
and use real C structs

My goal here, in case it isn't obvious, is to allow users to write
modules completely in Python, debug and develop in a great language,
and then recover the speed through compilation.  This has a side
benefit in that Jython and PalmPilot-thon and Embedded-processor-thon
can all use the Python variant of your code and if you run on a "real"
machine you get the speed.

Hope this is useful stuff.  If you want the code, drop me a line and
I'll pass it along.  I'm hoping that Eric will let me integrate this
directly with weave.  I think it is good enough to use out of the
box (since it doesn't barf at every opportunity), but it will take
some shaking out to get it to really work and this can only happen
with users!  Let me know who you are...

Pat