[Distutils] Wanted: ideas for using distutils with preprocessors/dependencies

[Jack Jansen]

I'm currently looking at integrating bgen with distutils.

Bgen is a little-known part of the Python core distribution: it is  
similar to swig, in that it generates C extension modules. In some  
respects it is more powerful than swig, the main one being that it  
reads standard C .h files in stead of adorned .i files.  
(Incidentally, this is also the reason it's part of core python, it  
is used to generate the MacOS API modules, so these are (almost)  
automatically updated when Apple adds new functionality. At least,  
that was true under MacOS9 and will be again when I get my act  
together:-). But bgen has a lot of disadvantages when compared to  
swig, the main one being that it is a rather fearsome tool to try and  
master. Integration with distutils is one of the things I want to do  
to lower that barrier.

But now I'm at a loss as to how to proceed. I had a look at how swig  
is integrated into distutils, and I don't really like it, it smells  
like a hack. And, according to the comments in the source and the  
manual, the author agrees with me:-) Swig support is basically done  
in the build_ext command, by filtering out all ".i" files in the  
source file list very early in the process, running swig on them, and  
replacing them by the .c or .cpp equivalents.

I can see various ways of adding bgen support, but I'm not sure which  
one is the best one, and/or whether there are other options. So I'd  
be interested in hearing what other people think, and how other  
packages have added a preprocessor to distutils.

There's a fair amount of Python code needed to drive bgen, at least  
for interfaces to complex APIs (bridging C types to Python, handling  
callbacks, how to parse the specific .h files for this API, etc).  
Currently that code is in two .py files but it will be put in a  
class, probably modeled somewhat after Extension (but having C/C++  
source files as output in stead of dynamic extension modules).

What I don't know is how I'd connect this to the Extension object  
that will create the extension module. Ideally I'd like the bgen  
process to be optional. In other words, the distribution packager has  
three options: (a) include the bgen C output in the distribution and  
don't run bgen unless the end users specifically asks for it; (b)  
include the bgen C output but only run bgen if the normal timestamp  
dependencies require it; or (c) always run bgen.

But it doesn't seem the Extension object currently has any support  
for such make-like chaining, and I'm not sure how to add it. One way  
would be to allow non-strings in the sources argument, and do  
something smart there. A similar mod could be used for libraries and  
extra_objects to allow chaining there too. Another way would be to  
add a "dependencies" argument, where those dependencies are objects  
that get run early, and can add their results to sources, libraries  
and extra_objects. I think this latter solution is probably better,  
as such a dependency object could modify multiple arguments of  
Extension in one fell swoop. As a somewhat contrived example, an  
"OptionalJPEGSupport" dependency could check whether the relevant  
libraries and include files are available to enable JPEG support in  
an imaging package, and then add the right source files, libraries,  
defines, library paths and include paths to the relevant Extension  
arguments.

But all of this is made quite a bit more difficult (I think) by the  
fact the Extension doesn't really do anything, it's only a container  
and all the logic is in build_ext. Maybe I should follow the paradigm  
set by "build_clib", and add a "build_bgen" command with build_ext  
picking up the results? And maybe there are better solutions that I  
haven't thought of yet?
--
Jack Jansen, <Jack.Jansen at cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma  
Goldman