[SciPy-dev] problems with numpy.setuptools...

[David Cournapeau]

David M. Cooke wrote:
>
>
> When you mention the 'f2c' library, you mean the Fortran intrinsic and
> runtime libraries? We try to use the Fortran compiler for linking, so
> knowing those tends not to be needed.
But what if we want to use C++ code in the future ? Then you have no 
choice but using the C++ compiler. But more to the point, this was an 
example. All AC_F77_* macro (or AC_C_*, etc...) are much, much more 
solid than distutils. And this is by design.
>
> For comparision, we've got 2348 lines of source in
> numpy/distutils/fcompiler/, and fortran.m4 included with autoconf is
> 1234 lines. 
To make it clear: I was not talking about using autoconf (which for once 
does not work on windows), but about the concept that almost everybody 
agrees autoconf is good at, that is working on a new platform not 
foreseen by the developer. And this is due to a fundamentally different 
way to look for platform capabilities.

We have a huge advantage compared to autoconf: python vs m4 + shell. 
Most of the lines in fortran.m4 are difficult because autoconf has to 
deal with weird shells, antic platform bugs, and because of m4 and its 
antic quoting rules.
> However, fortran.m4 mostly only deals with finding a Fortran
> compiler, finding flags for the libraries mentioned above, and name
> mangling (and nothing about optimisation flags). SCons also does its
> own handling of individual tools in much the same way as
> numpy.distutils (with, again, no optimisations).
What do you mean by no optimizations ? Dealing with facilities provided 
by scons and dealing with capabilities are two different things, IMHO, 
because I trust scons enough to work on what is supported. But for 
example, when distutils is looking for something, it is always 
hardcoded: looking for a library ? Distutils looks for a file, autoconf 
is looking for something usable by the linker. This makes a huge 
difference concerning the robustness (and the maintenance).
>
> Now, one design point that I think distutils gets wrong is to use
> lists for describing compiler options: the description of a compiler
> (or any of the command line tools used) should be an object, that
> 'knows' how to turn, say, a .c file into a .o file (something like
> scons' Builders, I think). We have some tools where the pattern
> EXECUTABLE_NAME + SOME_OPTIONS + file.c + MORE_OPTIONS + file.o +
> EVEN_MORE_OPTIONS doesn't work (most notably, linking shared libraries
> on AIX must be handled specially). The CCompiler and FCompiler classes
> are too high-level for this, as they contain knowledge about multiple
> tools.
This is something that scons seems to make right.
>
> Fiddling compiler options shouldn't be too hard, espicially for the
> Fortran compilers; it's usually pretty obvious where to fiddle for
> those. But other things (C compilers, linking), I agree, could be
> easier.
The point is, did it ever happen that somebody who had a problem with a 
compiler flag solved it by himself ?

>
> Could you detail your problems? system_info.py needs to be cleaned up
> (removal of repeative code, for instance) and finding libraries
> standardised (for instance, there should always be an environment
> variable and/or distutils key for things system_info wants to find,
> that will override system_info).
system_info is looking for files, which is not the right way in my 
opinion. For example, in audiolab, my system_info subclass is 70 lines 
(http://projects.scipy.org/scipy/scikits/browser/trunk/audiolab/setup.py), 
all this to find one library ! In scons  (or autoconf, or any other 
build tool I know), this is one line. Maybe I am doing things wrong, the 
code started when I knew nothing about python and just copied what I 
found in system_info.py, dunno. But the code in system_info does not 
seem to imply this is simple or robust.

This is really striking for me: I have no problem with complicated code, 
if it matches the complexity of the task it is trying to solve. But to 
me, distutils makes anything trivial complicated (again, just talking 
about the things related to compiled code).
>
> Blech, I hate the fact that scons doesn't respect my PATH. That means
> the compiler it may find is not necessarily the one that I would find
> at my command line. Makes bugs more obscure.
I fundamentally disagree on this one. But anyway, we don't even need to 
agree, since you can easily import the variables you want. So we can 
make this an option (which has always been my intention anyway), but by 
default, not importing anything is more robust for deployment, no ? 
People who have non standard tools or tools in non standard locations 
are the ones who should make the effort, not the contrary IMHO.
>
> Not if I steal stuff from scons ;-)
I doubt you can "steal" things from scons, for the very reason you 
described.
>
> (Or, for that matter, if I actually *had* any time to hack on it,
> which I don't right now.)
>
> Speaking of which, if the scons developers would make their code an
> actual Python package (for one thing, install it by default in
> site-packages/, and try to keep interfaces constantish), I would
> support using scons modules in numpy.distutils. From the looks of it,
> it's nicely laid out, with a good set of abstractions. Instead,
> they've set it up so that scons must be in control, through SConstruct
> files. 
I think this is partly due to the fact that scons support any python 
starting from 1.5.2, and partly to the fact that scons is not imperative 
(in fact, this is one key difference with waf AFAIK). But scons has a 
package scons-local, which can be used as a drop in (in fact, that's 
what I did in numpy.scons branch: in theory, we could make our 
modification there), which somewhat alleviates the problem.
> Those have always had a faux-python feel to them for me: there
> are names injected into them (i.e. Environment) that don't come from
> an import statement, and the sharing of variables between SConscript
> files using the Export function is quite unpythonic.
I would agree in theory, but compared to python distutils, which one do 
you think people can understand more easily ? For once, distutils (the 
one from python) adds many members attributes at runtime, in many 
functions (not just __init__); this combined to the fact that it is 
mostly not commented + not documented + not maintained (nobody to ask 
this, nothing on the internet), makes it harder for me to understand 
than I ever did with the autotools, which are not known for their user 
friendliness.
>
> Those are the big three. If it builds on those, we're probably 80% of
> the way there.
>
> Of course, it's the other 20% that'll take 80% of the work, as usual.
I agree that making the 20 % will take the most time (anybody with at 
least some experience with cross platform building would, I guess). But 
I think this will be much easier than with distutils.
>
> We can see what we can do to make it easier to get at those
> properties. For instance, it would be possible to separate the Fortran
> compiler detection/option-setting more from the distutils parts.
For example, one thing which caused me problems is to get the name of 
the compiler: for unix compiler, just using the member compiler[0] gives 
me the executable, from which I have a dict to convert to scons names. 
But MVSCCompiler does not have such a member; and to find the path of 
the executable, *sometimes*, you need to call initialize (which of 
course is not documented). This is not just unpythonic, this is just 
really bad code; what's the point to have a base class for compiler if 
this does not even have a common way to get the compiler name ?
>
>> To start building numpy itself with it would require more work (mostly 
>> tests for fortran / C abi), but with the help of people willing to help, 
>> I don't see major problems on this side, since scons provide a framework 
>> for autoconf-like testing. Also, this will certainly use some facilities 
>> of numpy.distutils (basically, for a first prototype, scons would mostly 
>> replace system_info.py, command/build_clib.py, command/build_ext.py).
>
> If you have the time, I think it's worth a try. Unfortunately, I don't
> have much time to contribute to it.
>
> Basically, my opinion is not to replace numpy.distutils with scons
> (for the simple reason of interacting nicely with distutils).
Again, never had the intention to totally replace distutils: distutils 
would still drive the whole build. Ideally, if it works out, I intend to 
replace the build_ext and build_clib commands; it would still works by 
calling setup.py build, it would still put everything in build, and 
install using --prefix, etc... This would be totally transparent to the 
user (except maybe for setting compiler at command line, and even, this 
may be doable to stay 100% backward compatible at this level). You can 
take a look at openalea, who did the exact same thing:

http://openalea.gforge.inria.fr/wiki/doku.php

Everything else seems to work quite well in distutils, no ?

cheers,

David