[Distutils] Improving distutils vs redesigning it (was people want CPAN)

[David Cournapeau]

On Thu, Nov 12, 2009 at 3:04 PM, Lennart Regebro <regebro at gmail.com> wrote:
> 2009/11/12 Robert Kern <robert.kern at gmail.com>:
>>> The discussion is pointless. If you want something better than distutils,
>>> do it.
>>
>> David's working on it:
>>
>> http://pypi.python.org/pypi/numscons
>
> "Enable to use scons within distutils to build extensions"
>
> I'm confused now.

The idea is that numpy.distutils adds a scons command, and if your
setup.py contains something like:

add_sconscript('SConscript')

It will call scons with enough options from distutils so that you can
build extensions from scons instead of distutils (it replaces all the
build_* commands except build_py, basically).

Numscons itself adds things like builders to handle python extensions,
so that you can do things like:

env.DistutilsPythonExtension('foo', source=['foo.c'])
env.DistutilsCtypesExtension('c_foo', source=['c_foo.c'])

Those CamelCase functions are called builders in scons parlance, and
any builder starting with Distutils* is such as it will put things
where distutils expects them (which is suprisingly difficult).

If you call python setup.py scons, you then get your extensions in
e.g. build/lib.win32-2.6/, and then all the bdist_wininst, etc... work
as expected. It solves our build issues beautifully:
 -  adding support for new compiler, new builder, new configuration
check, etc... is more pleasant than with distutils
 - the configuration framework is much more robust: you can check
things like type size, available header, available libraries and
functions in a cross platform way.
 - parallel builds works well
 - any change in one source file automatically trigger a new build of
the concerned files only (it goes as far as only relinking if you
change your library path).
 - changing compiler flags with CFLAGS works as expected.

But it has significant drawbacks, which means it is only useful as a
build/developer tool. This is mainly because scons and distutils
cannot interact 'bidirectionally' (both scons and distutils fault). In
particular:
 - source files added in scons are not automatically added to the
FileList object used for sdist production
 - changes in python files are not handled
 - we cannot integrate things like building a full .dmg or .msi from scons
 - we cannot build compatible eggs from scons because there is no
public API to do so

Most of the hard work in numscons is a 1000 LOC subpackage if I don't
count all the fortran stuff. Of course, it reuses scons, so that's
cheating. But note that scons has no knowledge whatsoever of python
extensions, and numscons can build extensions on windows (both 32 and
64 bits), linux, mac os x, solaris, freebsd, including debug
extensions. That's one of the reason why I find the claim of distutils
having a deep and large knowledge hard to believe.

David