[Numpy-discussion] Waf or scons/numscons for a C/Fortran/Cython/Python project -- what's your recommendation?

[David Cournapeau]

On Sun, Jan 17, 2010 at 4:12 AM, Kurt Smith <kwmsmith at gmail.com> wrote:
> My questions here concern those familiar with configure/build/install
> systems such as distutils, setuptools, scons/numscons or waf
> (particularly David Cournapeau).
>
> I'm creating a tool known as 'fwrap' that has a component that needs
> to do essentially what f2py does now -- take fortran source code and
> compile it into a python extension module.  It uses Cython to create
> the extension module, and the current configure/build/install system
> is a very kludgy monkeypatched Cython.distutils and numpy.distutils
> setup.py script.  The setup.py script works for testing on my system
> here, but for going prime time, I dread using it.  David has made his
> critiques of distutils known for scientific software, and I agree.
> What's the best alternative?

The best alternative in the short term is no alternative: making sure
everything you need is incorporated in numpy.distutils. Otherwise, you
will have to recreate everything that distutils is doing: you will
have people who will demand egg, mac os x .mpkg, windows installers,
etc.. Basically what I am trying to do with toydist now - I don't mind
getting help there, though :)

I promised to add decent cython support in numpy.distutils for 1.5.0,
maybe we should see what we can do for fwrap at the same time.

I am also a bit unclear about what is needed exactly, and what would
be the workflow: I don't understand why fwrap should care about
packaging/deployment at all, for example.

>
> More specifically: what are the pros/cons between waf and
> scons/numscons for configure/build/install of a
> Fortran-C-Cython-Python project?

waf has no fortran support whatsoever, so you would need to add it.
Waf codebase is much better than scons, but there lacks some internal
documentation. There were some things that I did not manage to do in
waf, because the internal API for scanning/dependency injection was
not very clear to me (by scanning I mean the ability to scan the
source code to look for dependency, e.g. fortran modules, and by
dependency injection, I mean adding new targets to the DAG of
dependencies at runtime - again needed for fortran modules).

Basic handling of fortran compilation and fortran detection was
relatively easy in comparison.

The biggest drawback I see with waf is the lack of users: the only
significant project I know which uses waf is Ardour. OTOH, I believe
scons has deep structural problems, and only a few people can change
some significant parts of the code.

>
> Is scons capable of handling the configure and install stages, or is
> it only a build system?  As I understand it, numscons is called from
> distutils; distutils handles the configure/install stages.

Distutils only handles the installation - everything done within the
build_* command is done by the scons distutils command, and
configuration is done by scons as well. Scons configure mechanism is
very primitive - it uses a separate framework than the rest of the
tool, which means in particular that scons top notch dependency
handling does not work well for the configuration stage. waf is much
better in that aspect.

> Scons/numscons have more fortran support that waf, from what I can
> see.  The main downside of using scons is that I'd still have to mess
> around with distutils.

My main point should be this: whatever you do, you will end up messing
with distutils, unless you reimplement everything that distutils does,
be it waf, scons, etc... In the short term, adding things to
numpy.distutils is the easiest path.

Long term, I hope toydist will be a tool which will enable exactly
what you want: using a build system of your choice, and being able to
reuse existing code for installation/packaging to avoid recreating it
yourself. You will be able to create an exe/egg/pkg from a simple
package representation, every package will have a common interface for
the user independently of the internal build tool, etc...

David