[Numpy-discussion] Bitting the bullet: using scons to build extensions inside distutils ?

[Pearu Peterson]

On Sat, September 15, 2007 9:41 pm, David Cournapeau wrote:

>    Starting thinking over the whole distutils thing, I was thinking
> what people would think about using scons inside distutils to build
> extension.

Are you thinking of Python distutils or numpy.distutils?
In the case of Python distutils, then I think this is a
wrong list to discuss this subject.

> The more I think about it, the more I think than distutils
> not being maintained,

numpy.distutils is being maintained.

> and numpy/scipy building needs being much more
> complicated (at least different) than usual python extension, trying
> to circumvent distutils problems is an ever ending fight.

why would be scons be different when new architectures and platforms
that need to be supported are developed now and in future?

> Scons, being
> developped as a Make replacement, can do all we would like to be able
> to do with distutils, including:
>  - building shared or static libraries, with dependencies (knows how
> to do it on many platforms).
>  - can build each object file independently (e.g different compiler
> options)

IMHO, this is the only important feature that distutils is missing
and would be difficult to add to numpy.distutils because of distutils
design.

>  - is much much friendlier than distutils.
>  - can handle external tools like swig, etc...

numpy.distutils can handle swig, psyco,.., also, it can generate
sources on-fly. It's not so difficult to add support for some
external tool to numpy.distutils.

>  - have basic facility to look for libraries (ala autoconf. By basic,
> I mean it is far from being as complete as autoconf, but is much
> better than distutils).

Can it achive the same as numpy/distutils/system_info.py?

> Scons has also the following advantages:
>  - written in python, can be distributed with numpy (by this, I mean
> AFAIK, license-wise, it is ok, and its size is not big): does not add
> additional dependency.
>  - can be called within distutils quite easily.

Scons does not have Fortran compiler support, at least not as advanced
as numpy.distutils.fcompiler provides, afaik. I think this is the main
reason why numpy.distutils cannot be replaced with scons without
having much of effort adding this feature to scons.

> That is, I don't see big disadvantage to use it with distutils. It
> would give use some wanted features out of the box (building
> extensions based on ctypes, much friendlier way to customize building
> option).

I agree that distutils is a difficult code. But over the years
lots of fixes and useful features have been implemented in numpy.distutils
that has made using numpy.distutils easier and less errorprone
than Python distutils. I don't even remember the last serious
issue with numpy.distutils what comes to building scipy or numpy
packages. Adding new features is another story of course.

> There are some things I am not sure about :
>  - how to build python extension with it: this is of course mandatory
>  - what is required for a "bi-directional" communication with
> distutils: for this to work, distutils needs to be aware of what scons
> builds (for things like bdist to work, for example).

It's fine to add new commands to numpy.distutils.

> There is no question this will require some work. But anyway, my
> feeling is there is a need to improve the distutils thing, and I feel
> like this may be an easier path than patching over distutils
> defficiencies. I know scons quite a bit, and am willing to develop at
> least a prototype to see the feasibility of the whole thing.
>
> But before starting, I would like to know whether other find the idea
> attractive, dumb, is a waste of time, etc...

I think the idea of replacing distutils with some other tool which
would be easier to use and to extend has brought up every one or two years.
I think it hasn't happened because it would require not just some
work but lots of it. Whatever people say about distutils, it is still
a good tool that has lots of know-how in it for doing its job in a large
variety of platforms we have today.

But if you think that you can make scons to support the numpy.distutils
features that are important for building numpy and scipy, then go for it.

Pearu