[Numpy-discussion] scipy.scons branch: building numpy and scipy with scons

[David Cournapeau]

Robert Kern wrote:
> David Cournapeau wrote:
>> Some of the most interesting things I can think of which work with scons:
>>     - you can control fortran and C flags from the command line: CFLAGS 
>> and FFLAGS won't override necessary flags, only optimization flags, so 
>> you can easily play with warning, optimization flags. For example:
>>
>> CFLAGS='-W -Wall -Wextra -DDEBUG' FFLAGS='-DDEBUG -W -Wall -Wextra' 
>> python setupscons build
>>
>> for debugging will work. No need to care about -fPIC and co, all this is 
>> handled automatically.
>
> Can I override the flags which are handled automatically without modifying
> numpy? I just spent much of last night trying to get Intel Fortran on OS X
> working, and I had to dive into numpy.distutils.fcompiler.intel a lot. This is
> mildly acceptable, if irritating, for a numpy developer, but not acceptable for
> even a sophisticated user. Even if we were to keep our knowledge-base of Fortran
> compiler flags immediately up-to-date with every release of every Fortran
> compiler we follow, people will still be stuck with older versions of numpy.
>
> numpy.distutils' behavior of using LDFLAGS, for example, to completely replace
> the flags instead of extending them mitigated this, somewhat. It allowed someone
> to work around the stale flags in numpy.distutils in order to get something
> built. This is a hack, and it causes confusion when this isn't the desired
> behavior, but it worked.
>
> But can we do something better with scons? One option which would work both with
> scons and the current numpy.distutils is to provide something like
> LDFLAGS_NO_REALLY which replaces the flags and let LDFLAGS just extend the
> flags. That doesn't help the ultimate problem, but it makes the workaround more
> user-friendly. Another option is to have our Fortran compiler "knowledge-base"
> separable from the rest of the package. scons could try to import them from,
> say, numpy_fcompilers first and then look inside numpy.distutils if
> numpy_fcompilers is not found. That way, a user could download a fresh
> "knowledge-base" into their source tree (and possibly tweak it) without the
> burden of installing a new numpy.
>
First, let me briefly outline the basic flow of scons (for every 
package, I call scons; since they are totally independant, we just 
consider one package):
    - the distutils scons command find the compilers, their path, and 
set up the build directory, and then call scons with those parameters at 
the command line (scons runs in its own process). This is done in 
numpy\distutils\command\scons.py
    - I create an environment using the parameters given by distutils: I 
initialize scons CC, FORTRAN and CXX tools. Those tools define several 
flags, like -fPIC for shared objects, etc... this is done in 
numpy\distutils\scons\core\numpyenv.py (_GetNumpyEnvironment).
    - I customize the flags depending on the compiler used: this is done 
in numpy\distutils\scons\core\default.py. This is where the optimization 
flags, warning flags are set. I then merge those flags with scons 
environments, and those are the ones the user get (in SConstruct) in 
GetNumpyEnvironment.

So you really have two places where flags are set:
    - optimization and warning are set in the default.py. We could use a 
file to override those quite easily.
    - flags like -fPIC are defined at the tool level. This is done 
inside scons; for cases where it does not work, I use my own modified 
tools (in numpy\distutils\scons\tools: any tool in this directory will 
be picked up first, before the scons ones).

So to go back to your problem: if I understand correctly, what is needed 
is to update the scons tools. Since those are kept at one place, I think 
it would be safe to update them independently. But I don't understand 
exactly how this could be installed in the source tree without 
reinstalling numpy ? I think this is better than completely overriding 
the compilation flags, personally. But if you really want this 
possibility, I can add it, too, without too much trouble.

Normally, the way I define and use compilation flags should be flexible 
enough to enable several approaches. I implemented the ones I most 
needed, but other are welcome.

David