[SciPy-Dev] On pulling fwrap refactor into upstream SciPy

[Dag Sverre Seljebotn]

It's a good time to start a discussion on my Fwrap refactor of Fortran 
wrappers in SciPy. I'd appreciate feedback from anyone who has an 
opinion on whether the Fwrap refactor should be merged upstream and, if 
so, how it should be done.

My SciPy branch is now feature complete without using f2py 
(cblas/clapack missing and only numscons build, not distutils -- the 
latter is definitely coming soon). What is there instead of f2py is 
Cython files that call the Fortran code directly. The Cython code should 
have the same API as f2py and be backwards compatible. The Cython files 
were generated from the f2py pyf files by using Fwrap, but the generated 
files were subsequently modified (rationale below) -- I've not really 
been replacing f2py with Fwrap, it is more correct to say that I used 
Fwrap to replace f2py with Cython.

My branch is here:

https://github.com/jasonmccampbell/scipy-refactor/tree/fwrap/

(Currently only numscons build is changed, not distutils. Also I didn't 
check in generated C sources for now.)

I'll provide some more detailed information on specific 
spots/differences once it's clear whether this is accepted upstream and 
how review will be done. Overall:

  * ABI: Same as f2py (Fortran 77 using the same blatant assumptions 
that f2py did)

  * Wrapper code typically looks like this:

https://github.com/jasonmccampbell/scipy-refactor/blob/fwrap/scipy/interpolate/dfitpack.pyx

In addition there's dfitpack_fc.h, dfitpack_fc.pxd which now expose the 
raw Fortran ABI for use from C or Cython code.

  * In the linalg package, there's pyx.in files, which are templates in 
the Tempita language (http://pythonpaste.org/tempita/). E.g.:

https://github.com/jasonmccampbell/scipy-refactor/blob/fwrap/scipy/linalg/fblas.pyx.in

  * Cython is more verbose, but also more explicit, than f2py. I also 
think Cython is friendlier but I'm obviously biased. Two examples to 
contrast the two:

i) "check(m > k)" in f2py code is done like this in Cython:

if not (m > k):
     raise ValueError('Condition on arguments not satisfied: m > k')

So more verbose, but OTOH it is now trivial to modify the code to 
provide a custom exception message, something impossible in f2py.

ii) Reordering the arguments in Cython is done by simply reordering the 
arguments in the wrapper function. And custom code is simply inserted, 
writing it in Cython. Whereas in f2py you get things like this:

      callstatement {int 
i=2*kl+ku+1;(*f2py_func)(&n,&kl,&ku,&nrhs,ab,&i,piv,b,&n,&info);for(i=0;i<n;--piv[i++]);}
      callprotoargument 
int*,int*,int*,int*,float*,int*,int*,float*,int*,int*

So, a (highly biased) summary: f2py code is much briefer for simple 
things, but can get very hairy once one needs to do something 
non-trivial. And the wrappers in SciPy often are non-trivial, there was 
an incredible amount of "secret" tricks employed in the pyf files to 
customize the wrappers that are now spelled out explicitly in Cython code.

However, the resulting code is more verbose, and perhaps more difficult 
to quickly scan.

Dag Sverre