[Numpy-discussion] [Announce] numpy.scons , ALPHA version

[David Cournapeau]

Hi there,

    I've finally managed to implement most of the things I wanted for 
numpy.scons, hence a first alpha. Compared to the 2d milestone from a 
few days ago, a few optimized libraries are supported (ATLAS, Sun 
sunperf, Apple Accelerate and vecLib, Intel MKL).

Who
===

Outside people interested in numpy.scons, people who have trouble 
building numpy may want to try this. In particular, using MKL or sunperf 
should be easier (for sunperf, it should work out of the box if sun 
compilers are used). Also, compiling with intel compilers on linux 
should be easier.

DO NOT USE THIS FOR PRODUCTION USE ! I am interested in success (that is 
build and all the test pass) as well as failure.

Numpy developers: I would be interested in comments about the code:
    - the package are built with the setupscons.py/SConstruct
    - most of the new code is in numpy/distutils/scons

How to test
===========

Grab the sources:

svn co http://svn.scipy.org/svn/numpy/branches/numpy.scons

And then just do: python setup.py scons; python setup.py install. If you 
have multiple CPU/Cores, you may want to use the parallel build options:

python setup.py scons --jobs=N with N an integer (number of CPU/CORES + 
1 is a rule of thumb).

For non default compilers, use the --compiler and --fcompiler options 
(distutils names).

For the MKL, you should add a mkl option in the site.cfg. For ATLAS or 
sunperf, if it is not in standard places, it won't work (yet).

What works:
===========

The tested platforms have been tested:
    - linux (ubuntu x86) with gcc, g77 and ATLAS.
    - linux (x86 and x64) with gcc, without BLAS or LAPACK.
    - linux (ubuntu x86) with gcc, g77, and mkl (9.1.023).
    - linux (ubuntu x86) with intel compilers.
    - windows (32 bits) with mingw.
    - windows (32 bits) with VS 2003.
    - solaris (solaris studio express 12) with sunperf on x86.
    - Mac Os X Intel with Accelerate framework.
    - People reported basic success on IRIX, too.

What does NOT work
==================

One important target missing is windows 64, but this should not be too 
difficult to solve.

There are still many corner cases not yet solved (in particular some 
windows things, most libraries path cannot yet be overriden in 
site.cfg); also, I do not attempt to be 100 % backward compatible with 
the current build system (that for a given environment, you won't 
necessarily have the same build options). Some things are not yet 
supported either (overriding libraries with env variables, for example, 
is not supported).

There is no real doc yet (I will do this once the basic public API is 
stabilized).

Details
=======

Library checkers
----------------

The main difference compared to the second milestone is the support for 
libraries checkers (think BLAS, Atlas, etc...). There is now support to 
check for different libraries, with different compilers. This 
effectively replaces numpy.distutils.system_info. The way I've 
implemented those checks are fundamentally different compared to the 
distutils way: instead of looking for files, I look for capabilities at 
runtime.

For example, for mkl, instead of looking for libmkl.so, etc... I try to 
compile, link and run a small mkl program: if any of the step failed, 
then the mkl is not used. This means that I can detect config errors 
much sooner; also, since I use rpath, there is no need to set things 
like LD_LIBRARY_PATH and so on. If the path is set right in site.cfg, 
then everything is taken care of.

Also, as before, a general library checker for new libraries is 
available: look at basic examples in numpy/scons_fake/. In particular, 
there is a NumpyCheckLib which can be used to check a library with some 
symbols, whose paths can be overriden with a site.cfg.

Fortran support
---------------

The other big work has been on fortran support. Here too, the way I 
implemented it is different than numpy.distutils. Instead of harcoding 
flags depending on versions, I use code snipets, to get link options, 
mangling, and so on. For example, if you call the following in a Sconscript:

config.CheckF77Mangling()

This will try to get the mangling of the F77 compiler, and if 
sucessfull, will set a python function mangle, which returns the fortran 
name from the 'common' name. For example, for g77, mangle('dgemm') will 
returns dgemm_. Since the mangling is found at runtime, this can be used 
for not yet known compilers.

Next
====

This was quite a lot of work, and I wanted to get something working as 
fast as possible. As such, the code is not always clean. Once I am 
satisfied with the global 'feel' of the new build, I will sanitize the 
API and implement the doc. Still, I believe the code to be much easier, 
simpler and robust than distutils (everything, from fortran support to 
libraries checkers is less than 2000 lines of python code, including 
tests; of course, this is largely thanks to scons).

The thing I am the less sure about is how to combine checkers in 
meta-checkers: for example, I have checker for mkl, ATLAS, sunperf, and 
we sometimes want a BLAS checker, which would check all those as long as 
nothing is found. API-wise, I am not yet satisfied with the current way.

I also want to play nice with f2py and scipy (totally untested for now).