[Distutils] Extension Modules

[Greg Ward]

On 20 September 2000, Mike Olson said:
> We are working towards converting 4Suite over to use distutils.  We are
> almost there, but have run into a problem.  We needed to implement our
> own parser to generate some files using a script we call BisonGen.

Mike --

first off, I'm not clear which distribution you're talking about here:
BisonGen or 4Suite?  (Or something else?  I downloaded those two source
tarballs from your ftp site last night, so those are my only points of
reference right now.)

> This generates both C and py files.  Then we hand the C off to
> build_ext to get compilied.  The .so (or pyd) is installed fine.
> However, the additional py files generated are not copied into the
> installation directory.  Is there a simple way of doing this?

Where does BisonGen put the generated .c and .py files?  In the source
tree or the build tree?

You might consider doing all preprocessing (run SWIG, BisonGen, lex,
yacc, etc.) *before* you put together the source tarball.  This would
make source distributions bigger, but would require less funky custom
code to run on the installer's machine, away from your ability to debug.
Then your custom stuff could just be run by a custom "sdist" command.
(Or you could do it manually, before running the stock "sdist".)  This
would also reduce the number of bits and pieces that people have to
install merely to build 4Suite.

Note that there's no rule that says you can only have one setup script
per module distribution.  You might have a big complex beast to handle
your local build needs -- or you might use a Makefile for that, since
the Distutils' dependency analysis of .c files is very primitive (no
header file analysis, in particular).  For example, I have one
"official" setup script that I use to create real releases, and another
one for creating code snapshots.  (They also have different manifests.)
Only the "official" setup script gets distributed in official releases.

> Looking through the
> code, build_ext only uses Extension.name to generate a list of outputs. 

If you're trying to figure out who decides what gets installed, you
should be looking in the install_* commands -- specifically, install_lib 
for Python modules and extensions.  I think the code you're looking for
is:

    def run (self):
        [...]
        # Install everything: simply dump the entire contents of the build
        # directory to the installation directory (that's the beauty of
        # having a build directory!)
        if os.path.isdir(self.build_dir):
            outfiles = self.copy_tree (self.build_dir, self.install_dir)

IOW, the "install_lib" command should be driven entirely by what it
finds in the "build" tree.  (This is a general principle which all the
"install_*" commands should follow.)

Anyways, I'm having a hard time understanding what's going on.  Once I
have SWIG installed and can actually run the 4Suite build, hopefully
I'll get clued in.

        Greg