[Distutils] People want CPAN

[Pauli Virtanen]

ke, 2009-11-11 kello 21:42 +0100, Tarek Ziadé kirjoitti:
[clip]
> Do you mean, an Extension that would require several compilers ?
> 
> I was thinking of a one-to-one relation between an Extension and a
> compiler type, even if there are are multiple source files (in different languages)
> for this extension.
> 
> Meaning that *one* compiler would have to handle of those files. Does that fit
> the cython use case?

It might. The distutils Cython compiler would however need to be smart
enough to handle all languages used in the other source files by itself,
or somehow delegate this back to distutils (which still sounds doable).

Also, the f2py compiler does similar things. Ditto for Pyrex. How much
code duplication would be needed between these?

Source file preprocessing / generation is one thing that also comes
useful. For example, Numpy has its own templated C code generator file
format, and also generates some of the source files automatically from
scratch during the build. These are compiled and linked as a part of
ordinary C extension module.

Currently, as I understand it, numpy.distutils does source generation in
a separate build_src command.

> > Also, the option of determining the necessary compiler and compiler
> > options from file extensions does not always work. For example, Fortran
> > 90 files come in two flavors, fixed-form and free-form, which often are
> > not differentiated by the file extension. However, they may require
> > different compiler flags to compile. (Hopefully, however, nobody uses
> > the fixed-form for F90 any more...)
> 
> How do they do then? Is file extensions, inline options in setup.py,
> and the environ, are enough?

numpy.distutils's Fortran compiler class reads the file and tries to
analyze whether it's fixed-form or not. And chooses appropriate compiler
flags. This is sort of a special "worst case" scenario for compiler
selection, of course.

    ***

Just to throw some wild, perhaps obvious, and definitely unasked-for
ideas in the air (especially as I can't promise I can give any sustained
help here :/ ):

I suppose one option would be to factor *everything* related to
extension module building into build_ext (and abolish build_clib): the
rest of distutils would just say to build_ext

        """
        Here's some info about the environment:

        dict(python_lib_path=/usr/lib/..., optimize=yes/no,
             python_lib_name=..., python_includes=/usr/include/...,
             install_prefix=/usr/lib/python2.6, ...
             ..., python_extension=..., build_temp_dir=...)

        Look the rest up from sysconfig.

        Please build any extensions and data files you like,
        and tell the file and directory names where you placed them
        and where (relative paths) they should go.
        """

Information needed how to build each component would be passed to
build_ext subcomponent directly from setup.py or from a config file.
Now, perhaps it is already like this --- I don't really know the
internals of distutils --- and the build subcomponent is insulated from
the others. In any case, something like this could make refactoring the
build system easier.

I think this idea quickly boils down more or less to David's idea about
a pluggable build system -- implementing a good one takes a lot of work,
so it might make sense to refactor distutils so that it would be
possible [1] to use some of the existing ones (scons, waf, whatever,
heck, even autoconf+make). The *default* build system could be a simple
one, and backwards compatible. Especially so, since it seems to me that
building extension modules is orthogonal to what distutils and
setuptools want to do -- package and install Python modules in a
Python-specific way, not really worry about how to properly call
different compilers on obscure platforms.

Anyway, even in the case pluggability is a bad idea, refactoring the
build system out from the rest of distutils might make sense.

.. [1] possible and easy -- I understand numpyscons is a stab
       at the possible, but it sounds like it was not easy to do.

-- 
Pauli Virtanen