[Distutils] [Fwd: Re: Annoucing distribute project]

[Phillip J. Eby]

At 03:34 PM 9/26/2008, Ian Bicking wrote:
>I'm not sure I agree with removing setup.py, though I dislike how it 
>functions currently -- that you kind of have to poke it from the 
>side to get basic information out of it, instead of something more 
>declarative.  In some ways it's the distutils/setuptools interface 
>that most people actually use -- pyinstall for instance only calls 
>setup.py in subprocesses to install the package, and never touches 
>it directly.  So anything that acts fairly similar to the current 
>setup.py's will be compatible.  This is part of why I don't think 
>the backward compatibility issue is as difficult as Guido suggests.
>
>Though... really if pyinstall could easily detect a new source 
>format and only setup.py with the old source format, it wouldn't be 
>hard to support both.  There does need to be room for custom code, 
>but mostly for the build/compilation process.

The main reason to drop setup.py as the interface is to prevent 
people writing their own crazy command line "parsing" (e.g. probing 
sys.argv directly) and from trying to install stuff or do other 
things unconditionally in code that shouldn't be doing anything but 
configuration.  Having a 'configure.py' or some such would be a clear 
break with the past in this respect.

That is, a configure.py script would be there if and only if you 
needed dynamic metadata or build-spec generation, and it shouldn't be 
designed to run as a user-run script, but instead be invoked in a 
restricted way by build and install tools.  It should also be 
required to be sandboxable, in the sense that it should write to no 
files outside the build area.  It should also be well-understood that 
it is not to unconditionally import dependencies, nor assume that 
failure to import a dependency means the dependency isn't present.

(And if you don't have any dynamic metadata requirements, the package 
metadata and build specs should be able to be defined in 
non-executable configuration files, with no need for a configure.py.)

IOW, the main reason (IMO) to drop backward compatibility is 
precisely to get rid of setup.py and thus all of the crazy hairy 
things people do with it that they've got no business doing (because 
they can't accomplish what they need any other way, without first 
becoming an expert at distutils internals).

Meanwhile, legitimate *build* extensions can and should be restrained 
to *building*, rather than installing.  We can define a build 
directory layout for installers to use for actual installation, 
including various platform-specific elements (such as icons/menu 
items, registry stuff, post-install scripts, etc.) which it's then 
the responsibility of individual installer programs to handle.

I guess that means that we would need to spec out:

1. Source tree layout
2. Build configuration files/build processing
3. configure.py
4. sdist format & metadata
5. build tree layout (for build code to build and installers to install from)

But the basic idea is that the stdlib supplies the 
build-and-distribute framework, package authors can have build 
extensions, but only install programs can actually install or 
uninstall anything.  The build layout should be flexible enough to 
allow things like dumping Debian or RPM or MSI metadata in there, for 
"install" tools (which might technically be platform bdist tools) to pull out.

This is a rather important separation of duties, because right now 
bdist tools are responsible for build management and tricking 
"install" commands in order to do their work.  This makes it too hard 
for people to write bdist tools.  (And you can't even *think* about 
custom install tools with the distutils now.)

These elements would make it possible for the distutils to be divided 
up into more maintainable slices, that are owned by a much larger 
group of people, with much lower probability of burnout.  Given a 
stable build layout, install tools aren't likely to change 
much.  Given a stable source and build layout, build tools aren't 
likely to change much.  The hairiest bits (extension/compiler stuff) 
will have to be in stdlib anyway, and are fairly isolated within the 
overall process.

(I also think we should encourage the model pioneered in buildout and 
setuptools of keeping build extensions as separate packages from the 
things they're used to build, but I could be convinced that making it 
*possible* (but not necessarily easy) to include a build extension 
within a package that's using it, could *perhaps* be a reasonable idea.)