[Distutils] Builders vs Installers

[PJ Eby]

On Mon, Mar 25, 2013 at 5:08 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> There's a longer-term issue that occurred to me when thinking about
> pip's role as a "builder" or an "installer" (to use Nick's
> terminology).
>
> As I understand Nick's vision for the future, installers (like pip)
> will locate built wheels and download and install them, and builders
> (like distutils and bento) will be responsible for building wheels.
> But there's an intermediate role which shouldn't get forgotten in the
> transition - the role that pip currently handles with the "pip wheel"
> command. This is where I specify a list of distributions, and pip
> locates sdists, downloads them, checks dependencies, and ultimately
> builds all of the wheels. I'm not sure whether the current idea of
> builders includes this "locate, download and resolve dependencies"
> function (distutils and bento certainly don't have that capability).

Yes, and to make things even more interesting, consider the cases
where there are build-time dependencies.  ;-)

I would guess that installing from sdists (and revision control) is
probably here to stay, along with the inherent coupling between
"build" and "fetch" functions.

Right now, the "build" side of setuptools fetches build-time
dependencies, but in the New World, ISTM that a top-level build tool
would just be something that reads the package metadata, finds
build-time dependencies, and then runs some entry points to ask for a
wheel to be spit out (or to get back a data structure describing the
wheel, anyway).

This part could be standardized and indeed could be just something
that pip does.

Another piece that hasn't seemed well-specified so far is what Nick
called "archiving" - creating the sdist.  Or perhaps more precisely,
generating an sdist PKG-INFO and putting the stuff together.

IMO, a good install tool needs to be able to run the archiving and
building steps as well as installing directly from wheels.  However,
although metadata 2.0 provides us with a good basis for running build
and install steps, there really isn't anything yet in the way of a
standard for sdist generation.

Of course, we have "setup.py sdist", and the previous work on
setup.cfg by the distutil2 team.  We could either build on setup.cfg,
or perhaps start over with a simple spec to say how the package is to
be built: i.e., just a simple set of entry points saying what
archiver, builder, etc. are used.  Such a file wouldn't change much
over the life of the package, and would avoid the need for all the
dynamic hooks provided by distutils2's setup.cfg.

In the degenerate case, I suppose, it could just be "pyarchiver.cfg"
and contain a couple lines saying what tool is used to generate
PKG-INFO.

On the other hand, we could draw the line at saying, pip only ever
installs from sdists, no source checkouts or tarballs.  I'm not sure
that's a reasonable limitation, though.

On the *other* other hand, Perhaps it would be best to just use the
setup.cfg work, updated to handle the full metadata 2.0 spec.  As I
recall, the setup.cfg format handled a ridiculously large number of
use cases in a very static format, and IIRC still included the
possibility for dynamic hooks to affect the metadata-generation and
archive content selection processes, let alone the build and other
stages.

But then, is that biasing against e.g. bento.info?  Argh.  Packaging
is hard, let's go shopping.  ;-)

On balance, I think I lean towards just having a simple way to specify
your chosen archiver, so that installing from source checkouts and
dumps is possible.  I just find it annoying that you have to have
*two* files in your checkout, one to say what tool you're using, and
another one to configure it.

(What'd be nice is if you could just somehow detect files like
bento.info and setup.cfg and thereby detect what archiver to use.  But
that would have limited extensibility unless there was a standard
naming convention for the files, or a standardized format for at least
the first line in the file or something like that, so you could
identify the needed tool.)