[SciPy-Dev] On pulling fwrap refactor into upstream SciPy

[Dag Sverre Seljebotn]

On 02/08/2011 10:58 PM, Pauli Virtanen wrote:
> On Tue, 08 Feb 2011 21:48:28 +0100, Dag Sverre Seljebotn wrote:
>    
>> On 02/08/2011 12:09 PM, Pauli Virtanen wrote:
>>      
> [clip]
>    
>>> If manual changes are necessary, can they be cleanly separated from
>>> the automatic boilerplate?
>>>
>>>        
>> Not code-wise, but history-wise.
>>
>> That is, I have several times changed the code generation of Fwrap,
>> regenerated the boilerplate, and put the manual changes back in (using
>> git).
>>      
> [clip]
>    

First: Are you sure you are looking at the right branch? There should be 
no *_fc.f files now (if so it is a bug), I did get rid of them.

I'm pushing to Jason's account. I guess I should remove scipy-refactor 
on my account...

https://github.com/jasonmccampbell/scipy-refactor/tree/fwrap

> Clever, that's doable with git. However, it's not very pretty, and I have
> a nasty feeling about mixing autogenerated code with manual code in the
> long term.
>    

The thing is, this isn't really different from f2py. The pyf files are 
*also* first generated automatically from Fortran files, and then 
manually modified.

So it is a change of language (and one may argue that pyf files are 
shorter and more declarative -- but also more full of ugly hacks to get 
around limitations). But it is not really a change of approach.

To be clear, I was hoping to rip out/remove pyf files eventually, and 
consider the Cython wrappers the "final" wrappers. If the Fortran code 
does change, one can either rewrite that part of the wrapper, or use 
git. Just like f2py.

> It however looks like that a viable plan [A] could be to replace all F2Py
> wrappers in Scipy with more or less manually written Cython wrappers. The
> wrapper code doesn't look *too* bad, and is in any case almost write-
> once. This would require factoring out common parts of the Fwrap code
> pieces (e.g. all the fw_* functions) to a reusable-within-Scipy Cython
> library. Some other common tasks for such a library: helpers for
> declaring ufuncs etc.
>    

That's a good point. To be honest, I wasn't sure where to put such 
cross-package code, since e.g. the f2py build does copy in the shared 
sources into every package in the build, and since there's been pushes 
now and then for breaking up the packaging. Creating such a shared 
"scipy.core" or "scipy.utils" seemed orthogonal to what I was doing *shrug*.

> Another plan [B] could be to try to improve Fwrap until it can produce
> compatible results directly from the .pyf sources, or some other mostly
> declarative definition. (This would probably be a nice boon for Fwrap
> itself, too.) Here, one would need to think to which degree this is
> possible at all; you probably have a pretty good understanding on this.
>    

Well, the pyf sources as is is a nogo for the .NET port as they contain 
some CPython-specific code. Augmenting the pyf files with inline Cython 
code didn't seem pretty. But even if we did, getting the amount of 
manual changes down to 0% from pyf files is realistically not going to 
happen, as I think nobody would be interested in doing it. Let's exclude 
that as a possibility.

Now, if one only considers CPython, you could stick with f2py. I'm not 
saying this merge *has* to happen (as I said initially I can see a case 
either way).

The thing is, pyf is declarative, but severely limited. This leads to 
hundreds of ugly hacks (many of them invisible to the untrained eye -- 
such as casting a real Fortran-side to a complex Cython-side by passing 
the address of a complex variable as a real pointer to Fortran), many of 
them intertwined with the exact f2py implementation.

So I guess the lesson I took from pyf files is that such a simple 
declarative language is *not* powerful enough for wrapping Fortran 
source (or at least, people weren't satisfied with 1:1 wrappers, but 
added lots of customizations to the wrappers -- you *could* do "ipiv += 
1" for the PLU factorization Python-side, but that's not what people 
did). And in light of that, I kind of looked at manually modifying 
Cython wrappers as a feature, not a mis-feature.

(Long-term, I'd much rather spend my time improving Cython so that the 
Cython wrappers can contain less boilerplate just because of more things 
being built into Cython. After all, if you wrap C code directly in 
Cython, there's much of the same boilerplate, and it should be easier.)

I could invent a totally new declarative language for Fwrap (or a 
dialect of pyf that fixed its problems). But that would have the same 
problems that pyf has currently (see Fabian's post -- dropping pyf is 
one less language to learn).

BTW, if Fortran sources change, you should be able to do

fwrap update mywrapper.pyx

and it will parse the configuration stored in there, reparse Fortran 
file, do a git blame, do proper branching, and tell you to merge.

> Continuing with stupid questions (although this one wasn't answered in
> your initial mail :) --
>
> What are those *_fc.f files for? It seems their purpose is to deal
> with assumed-shape arrays. However, unlike F90, AFAIK for F77 one should
> just assume that double precision :: a(10,10) is compatible with
> a(10*10). So as far as I see, all *_fc.f could be dropped.
>    

Yes, you got the wrong branch, sorry about that.

Dag Sverre