[SciPy-user] Low-level code integration discussion at scipy'03?

[Fernando Perez]

Prabhu Ramachandran wrote:

> AFAIK, the other problems with blitz++ and weave are:
> 
>  1. Blitz is not as fast as a hand coded inline C version.  More on
>     ugliness of code later.

 From my experience, this is definitely true.  Depending on how much one uses 
Blitz's abstractions, the overhead can be rather high.  This is one of the 
things I'd like to discuss with J. Cummings at Caltech, since he works on the 
blitz internals.

>  2. Blitz compile times are large.

True.

>  3. An extra layer of complexity for weave, one more package to keep
>     track of -- the blitz sources are bundled with weave.
>  4. With all due respect, not many C++ programmers (forget Python
>     programmers) would be able to understand/code anything with the
>     techniques used inside Blitz.  Agreed, this is transparent to the
>     user in the present context but its still worth noting.

True.  Using it, however, is a dream :)

> Here is my take.  I've always felt that there should be a way to
> seamlessly handle Python/C/C++ Numeric arrays using a standard API for
> each language.  I don't believe I've mentioned it anywhere though and
> I am not sure if such a thing exists or not.  Numeric/numarray offer
> the most commonly used, maintained and popular API for arrays for
> numerics in Python.  It would be best if a clean C/C++ API would
> exist.  I'm thinking of a higher level wrapper like class that gives
> the C/C++ user a nice API.  No re-implementation of the
> Numeric/numarray API should be neccessary.  If this were done, the
> following becomes easily possible:
> 
>  1. Its easy for C/C++ programmers to use the API and not have to
>     worry about translating their C code to Python.  The translation
>     should be easy since they are using the standard wrapper API that
>     uses the raw numeric API underneath.
>  2. Performance is not too much of an issue since there is no copying
>     of data back and forth between C/Python and the C++ layer would
>     most probably be entirely inlined.
>  3. Writing weave.inline code should be almost as easy as the blitz
>     version.  Agreed blitz is extremely easy to use but if we had a
>     fairly powerful wrapper API most of the common stuff could be
>     easily done via this API.  Agreed, getting slices to work would be
>     some work but if there is significant interest, this should also
>     be possible.  In any case the API could be made sufficiently
>     useable without the need for blitz conversions.
>  4. Compile times should drop since we aren't really looking to redo
>     blitz but simply provide a clean Numeric API for C/C++.

Well, I'm quite afraid that in order to get something like this done right, 
one would probably end up quite close to blitz (or something similar).  The 
big problem with getting an efficient, low-level class representation for 
arrays is avoiding temporaries and extra copies in array expressions.  As far 
as I know, the only way to achieve that is to use expression templates, a la 
blitz.  This is just hard.  So yes, perhaps there is a way to do this without 
blitz/ublas.  But if one ends up faced with the prospect of writing a full 
expression template system, might as well use one of the existing ones.

I am not saying that blitz has to be used.  It's just that from what I've 
seen, it seems to satisfy the following requirements:

- trivial to get a blitz array from a numarray -> no need to change anything 
in numarray's design to use it.

- it tackled the problem of expression analysis to avoid temporaries and extra 
copying.  The technique used was expression templates (very complex code), but 
I don't know of any simpler solution for this problem.

So if you think that it is possible to obtain something similar with much less 
work, and lower compilation times/simpler code, great.  Let's then have this 
discussion session at Scipy'03!  (hint to the organizers: drop us a word about 
scheduling options, or we can plan for a lunch/dinner/beers time for this :)

Best,

f.