[Numpy-discussion] For review: first milestone of scons support in numpy

[David Cournapeau]

Pearu Peterson wrote:
>
> Yes. I was considering the situations where distutils in not used at 
> all. May be it will be relevant in future..
>
If distutils is not used at all, there will be more difficult problems 
to solve than this one, then. I think it is safe to assume that 
distutils will stay there for quite some time, no ?

Note that I try hard in numpy/scons to separate the scons specific from 
the parts which are not specific to scons. If in the future, we decide 
to go on the waf route, or whatever, I hope that most of the logic would 
be easily reused: for example, parsing the fortran link output to guess 
the link options for C/Fortran is totally independant from scons (this 
has the nice advantage of making regression tests possible on all 
platforms).
>
> I would prefer the former solution where each package is totally
> independent. Let's first see what is the acctual performance and then
> maybe consider more complex solutions if the build is really slow.
I measured the initialization time of scons if called for every 
subpackage in numpy/: this adds less than two percents of total build 
time, so this is not as bad as I feared (specially since only a few 
package would need scons: pure python ones do not need them). And scons 
can also give some speed advantage, because it explicitely support in a 
totally safe way parallel builds.

So basically, I went to the same conclusion than you. I have started 
porting numpy.core to scons (by far the most difficult subpackage, at 
least a priori) following this route.
>
> These macros are fine. May be we need more such macros for F90 compilers
> but we can discuss about these when the corresponding issues will emerge.
Oh, I forgot to say it, but the macro would be implemented for both F77 
and F90 (and F2003 or whatever should we need it). As most of the logic 
is independent of the version used, this is not a problem.

>
> However, if you could also provide the macros
> that f2py uses then everything would work with no change in f2py part.
> f2py uses the following defintion of F_FUNC, for example:
>
I can of course change the name of the macro (this could be an option, 
even). I guess it is better to follow f2py convention than autotools 
here, in our case.
>
> system_info has a feature of providing architecture specific
> and optimization flags for fortran compilers. Could scons
> support such a feature?
scons uses the concept of Environment objects: they have sensible 
defaults (compiler/platform combination), but you can override anything 
you want, and they are used for compilation.

env = Environment()
env.AppendUnique(F77FLAGS = "-O3")
env.Program('hello', source = ['hello.f'])

Of course, in our case, we should provide some support such as instead 
of adding -O3, we have something like a FULL_OPTIMIZED variable which 
contains the optimization flags tweaked for a particular 
compiler/platform. Better, we can check that a given flag is actually 
supported. This is something I have not done yet, but that's the kind of 
things scons does really well, so I don't see any problems here.

The main problem right now is to convert all the checks in 
numpy/core/setup.py, which is the only non trivial setup.py file as far 
as I can tell in numpy. This will take a few hours to get it right on 
major platforms :)

cheers,

David

P.S If you have access to some fortran compiler you would like to check, 
you can check my bzr branch on launchpad (this is done outside numpy 
because some things may be merged to scons sources at some point):

https://code.launchpad.net/~david-ar/numpy.scons.support/dev

And test it by: scons CC=ccompiler F77=f77compiler F90=f90compiler. For 
example: scons CC=icc F77=g77 F90=ifort.

(If you change compiler, you may need to add the option --config=force).

This is a "fake" sconscript which tests the following:
    - does the fortran compiler works (e.g. can compile a dummy program)
    - find the dummy main
    - find the verbose flag (useful to find informations at runtime)
    - find the name mangling
    - find the necessary options for C/Fortran mix (only implemented for 
F77 for now).

There is a pretty good chance that you will be able to break it (in 
particular finding the options for C/Fortran: I have not yet implemented 
all the necessary parsing), but in this case, please provides me with 
the config.log (which logs all the configuration dones by scons).