[C++-sig] weave plus boost?

[eric]

Hey Ralf,

> > My main interest here is to see if there is anyway that
> > weave can benefit from the boost effort -- I'm almost sure
> > there is.  weave (www.scipy.org/site_content/weave) is a
> > tool for, among other things, inlining C/C++ within
> > Python.
> 
> What is the main purpose of weave? Where is it applied?

Right now, I think it is best applied for the following:
    
    1. Fast algorithms.  This is what I use it for most.
    2. Extending the capabilities of an extension library 
       without building an entire new extension.
    3. Handling problems that are more easily solved  in C/C++

> Is my impression correct that weave uses python to call the C/C++
> compiler? How do you address the issue of portability (custom
> compiler flags, choice of compiler, etc.)?

weave uses distutils to build extensions on the fly.  It makes a 
simple hacks to force the linker to be g++ instead of gcc when gnu
is used so that C++ files work appropriately.  It also works using 
Microsoft C++ compiler.  distutils has a method to specify compiler
flags on a global bases.  If you need per-file specific flags, this
isn't handled.  Switching compilers between MSVC and gnu is possible
on windows, but not on other platforms. The same compiler that was 
used to compile Python will be used to compile your extension -- 
distutils reads this information from a Makefile distributed with
with Python.  There are ways of customizing this, but it requires 
effort.

> 
> My view of the Python/C++ world is roughly this:
> 
> I use Python where I can (i.e. performance is not critical)
> and C++ where I must. Universally, the C++ code operates on
> larger blocks of data (in the simplest case a vector).
> Standard vector operations would be best covered by a
> vector library (ecological niche of NumPy, but ideally with
> true C++ vector types), more complex operations can easily be
> interfaced via Boost.Python (i.e. the interface code tends
> to be small compared to the code for the complex operation).

We have similar Python/C++ world views.  I use Python for pretty
much everything, but found that, even with Numeric, there were
many situations where I needed more speed for an algorithm.  This
is primarily for scientific applications, but also comes up in
other situations.  So, weave is designed to solve these problems
as easily as possible.

> How does weave fit into this picture?

I'll give a couple more examples that'll hopefully give you a better
picture.  For more examples, go here:

http://www.scipy.org/site_content/weave/tutorial.html
http://www.scipy.org/site_content/weave/python_performance.html

Laplace equation:

Here is an example of the inline code for solving the
Laplace equation.  It converts Numeric arrays to Blitz++ arrays 
(a C++ array library).  It is pretty simple to specify how Python
types are converted to C++ types, so if you have a vector library
you prefer (or STL or whatever), this is not hard.  The following
is a Python function with the expensive part of the algorithm
coded in C++ using inline():

def inlineTimeStep(self, dt=0.0):        
    """ Takes a time step using inlined C code -- this version uses
        blitz arrays.
    """        
    g = self.grid
    nx, ny = g.u.shape
    dx2, dy2 = g.dx**2, g.dy**2
    dnr_inv = 0.5/(dx2 + dy2)
    u = g.u
        
    code = """
           #line 120 "laplace.py" (This is only useful for debugging)
           double tmp, err, diff;
           err = 0.0;
           for (int i=1; i<nx-1; ++i) {
               for (int j=1; j<ny-1; ++j) {
                   tmp = u(i,j);
                   u(i,j) = ((u(i-1,j) + u(i+1,j))*dy2 +
                             (u(i,j-1) + u(i,j+1))*dx2)*dnr_inv;
                   diff = u(i,j) - tmp;
                   err += diff*diff;
               }
           }
           return_val = Py::new_reference_to(Py::Float(sqrt(err)));
           """
    # compiler keyword only needed on windows with MSVC installed
    err = scipy.weave.inline(code,
                             ['u', 'dx2', 'dy2', 'dnr_inv', 'nx','ny'],
                             type_factories = blitz_type_factories,
                             compiler = 'gcc',
                             extra_compile_args =
                             ['-O3','-malign-double','-funroll-loops'])
    return err

This is taken from a good comparison of several techniques made by Prabhu
Ramachandran:

http://www.scipy.org/site_content/weave/python_performance.html

In the end, inline() can get within 25% of a program written completely
in C/C++ on this problem for a 500x500 array.

Dictionary sorting:
Alex Martelli gives an example of returning the values of a dictionary 
in the sorted order of the dictionary keys in the Python Cookbook.

    def sortedDictValues3(adict):
        keys = adict.keys()
        keys.sort()
        return map(adict.get, keys)    
Alex provides 3 algorithms and this is the 3rd and fastest of the set. 
The C version of this same algorithm follows: 
       
    def c_sort(adict):
        assert(type(adict) == type({}))
        code = """     
        #line 21 "dict_sort.py"  
        Py::List keys = adict.keys();
        Py::List items(keys.length()); keys.sort();     
        PyObject* item = NULL; 
        for(int i = 0;  i < keys.length();i++)
        {
            item = PyList_GET_ITEM(keys.ptr(),i);
            item = PyDict_GetItem(adict.ptr(),item);
            Py_XINCREF(item);
            PyList_SetItem(items.ptr(),i,item);              
        }           
        return_val = Py::new_reference_to(items);
        """   
        return inline_tools.inline(code,['adict'],verbose=1)

The C++ version is opts out of using CXX's nice methods for indexing
Lists, etc. in the loop, because they didn't prove to be very fast.
The C version works for every key/value type that the original Python
function does, and runs about 2 times faster.

thanks,
eric



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20020120/631ae31c/attachment.htm>