[SciPy-Dev] On pulling fwrap refactor into upstream SciPy

[Dag Sverre Seljebotn]

On 02/02/2011 12:53 PM, Dag Sverre Seljebotn wrote:
> It's a good time to start a discussion on my Fwrap refactor of Fortran
> wrappers in SciPy. I'd appreciate feedback from anyone who has an
> opinion on whether the Fwrap refactor should be merged upstream and, if
> so, how it should be done.
>
> My SciPy branch is now feature complete without using f2py
> (cblas/clapack missing and only numscons build, not distutils -- the
> latter is definitely coming soon). What is there instead of f2py is
> Cython files that call the Fortran code directly. The Cython code should
> have the same API as f2py and be backwards compatible. The Cython files
> were generated from the f2py pyf files by using Fwrap, but the generated
> files were subsequently modified (rationale below) -- I've not really
> been replacing f2py with Fwrap, it is more correct to say that I used
> Fwrap to replace f2py with Cython.
>
> My branch is here:
>
> https://github.com/jasonmccampbell/scipy-refactor/tree/fwrap/
>
> (Currently only numscons build is changed, not distutils. Also I didn't
> check in generated C sources for now.)
>
> I'll provide some more detailed information on specific
> spots/differences once it's clear whether this is accepted upstream and
> how review will be done. Overall:
>
>    * ABI: Same as f2py (Fortran 77 using the same blatant assumptions
> that f2py did)
>
>    * Wrapper code typically looks like this:
>
> https://github.com/jasonmccampbell/scipy-refactor/blob/fwrap/scipy/interpolate/dfitpack.pyx
>
> In addition there's dfitpack_fc.h, dfitpack_fc.pxd which now expose the
> raw Fortran ABI for use from C or Cython code.
>
>    * In the linalg package, there's pyx.in files, which are templates in
> the Tempita language (http://pythonpaste.org/tempita/). E.g.:
>
> https://github.com/jasonmccampbell/scipy-refactor/blob/fwrap/scipy/linalg/fblas.pyx.in
>
>    * Cython is more verbose, but also more explicit, than f2py. I also
> think Cython is friendlier but I'm obviously biased. Two examples to
> contrast the two:
>
> i) "check(m>  k)" in f2py code is done like this in Cython:
>
> if not (m>  k):
>       raise ValueError('Condition on arguments not satisfied: m>  k')
>
> So more verbose, but OTOH it is now trivial to modify the code to
> provide a custom exception message, something impossible in f2py.
>
> ii) Reordering the arguments in Cython is done by simply reordering the
> arguments in the wrapper function. And custom code is simply inserted,
> writing it in Cython. Whereas in f2py you get things like this:
>
>        callstatement {int
> i=2*kl+ku+1;(*f2py_func)(&n,&kl,&ku,&nrhs,ab,&i,piv,b,&n,&info);for(i=0;i<n;--piv[i++]);}
>        callprotoargument
> int*,int*,int*,int*,float*,int*,int*,float*,int*,int*
>
> So, a (highly biased) summary: f2py code is much briefer for simple
> things, but can get very hairy once one needs to do something
> non-trivial. And the wrappers in SciPy often are non-trivial, there was
> an incredible amount of "secret" tricks employed in the pyf files to
> customize the wrappers that are now spelled out explicitly in Cython code.
>
> However, the resulting code is more verbose, and perhaps more difficult
> to quickly scan.
>    

Forgot to say one thing: The fwrap branch does NOT depend on other parts 
of the refactoring effort; in particular it doesn't need NumPy 2.0. So 
it can be reviewed and merged independent of the rest...probably best 
time is right after a release to make sure it gets some more testing...

Dag Sverre