[Distutils] Builders vs Installers

[PJ Eby]

On Mon, Mar 25, 2013 at 11:35 PM, Daniel Holth <dholth at gmail.com> wrote:
> The problem we are solving first is not "setuptools", it is simply
> that right now Python programs are usually installed by downloading
> and then running a program that performs the actual install. Instead,
> we're going to let the installer do the actual install.
>
> The funny thing is that all the same things happen. We just move the
> responsibilities around a little bit, things like Bento or distutils2
> don't have to implement their own installer code, and we don't have to
> worry about that code doing bad things like messing up our systems or
> accessing the Internet. The installer might do new clever things now
> that it really controls the install.
>
> There are a tremendous number of things you can do from there, mostly
> undeveloped, including decoupling the rest of the packaging pipeline
> all the way down to the humble sdist, but we don't really need to
> change the pip user interface. All the same things will continue to
> happen, just rearranged in a more modular way.

I'm not sure what you're trying to say here; all the above was assumed
in my post, as I assumed was implied in Paul's post.  I was talking
about the *how* we're going to accomplish all this while still
supporting building from pure source (vs. an sdist with a 2.0
PKG-INFO).

More specifically, I was hoping to move the discussion forward on
nailing down some of the details that still need specifying in a PEP
somewhere, to finish out what the "new world" of packaging will look
like.


> The MEBS "ultimate punt" design is that you would iterate over build
> plugins to recognize an sdist. The sdist would be defined as anything
> recognized by a plugin.

I'm only calling it an "sdist" if it includes a PKG-INFO, since right
now that's the only real difference between a source checkout and an
sdist.  (That, and the sdist might have fewer files.)  So, source
checkouts and github tarballs aren't "sdists" in this sense.  (Perhaps
we should call them "raw sources" to distinguish them from sdists.)

Anyway, the downside to a plugin approach like this is that it would
be a chicken-and-egg problem: you would have to install the plugin
before you could build a package from a raw source, and there'd be no
way to know that you needed the plugin until *after* you obtained the
raw source, if it was using a freshly-invented build tool.


> It's probably more practical to at least name
> the preferred build system in a very minimal setup.cfg. This is still
> a little bit ugly. In a normal "new style" sdist, you may be able to
> trust the PKG-INFO file, but when building from source control you
> can't, would need to inspect setup.cfg, and ask the build system to
> refresh the metadata.

Right.  Or technically, the "archiver" in Nick's terminology.
Probably we'll need to standardize on a config file, even if it makes
life a little more difficult for users of other archiving tools.

OTOH, I suppose a case could be made for checking PKG-INFO into source
control along with the rest of your code, in which case the problem
disappears entirely: there'd be no such thing as a "raw" source in
that case.

The downside, though, is that there's a small but vocal contingent
that believes checking generated files into source control is a sign
of ultimate evil, so it probably won't be a *popular* choice.

But, if we support "either you have a setup.cfg specifying your
archiver, or a PKG-INFO so an archiver isn't needed", then that would
probably cover all the bases, actually.