[Distutils] Builders vs Installers

[PJ Eby]

On Wed, Mar 27, 2013 at 10:09 PM, Donald Stufft <donald at stufft.io> wrote:
> Right, which is why I think the ability to install from a raw source is a good feature for an installer, but not for the dependency metadata.

Sure - I never said the dependency metadata had to be able to say
*where* you get the raw source from, just that the tool for resolving
dependencies needed to be able to process raw source into something
installable.

> Following that we just need a standard way for a raw source tarball to declare what it's builder is, either via some sort of file that tells you that, or a build script , or something along those lines.

Yep.  Static configuration is a *must*, here, though, as we want to
move away from arbitrary setup script writing by package authors: in
general they are really bad at it.  A lot of setuptools' odd
build-time features (like sandboxing) exist specifically because
people write whatever crap they want in setup.py and have zero idea
how to actually use/integrate with distutils.

One interesting feature that would be possible under a
configuration-based system is that you could actually have an
installer with a whitelist or blacklist for build tools and
setup-requires, in order to prevent or limit untrusted code execution
by the overall build system.  This would make it slightly more
practical to have, say, servers that build wheels, such that only
tools the servers' owners know won't import or run arbitrary code are
allowed to do the compiling.  (Not that that should be the only
security involved, but it'd be a cool first-tier sanity check.)

(Interestingly, this is also an argument for having a separate
"tests-require-dist" in metadata 2.0, since testing tools *have* to
run arbitrary code from the package, but archivers and builders do
not.)


Nick wrote:
>> However, as I've said elsewhere, for metadata 2.0, I *do not* plan to
>> migrate the archiving or build steps away from setup.py. So "give me
>> an sdist" will be spelled "python setup.py sdist" and "give me a wheel
>> file"  will be spelled "python setup.py bdist_wheel".

Works for me.  Well, sort of.  In principle, it means you can grow
next generation build systems that use a dummy setup.py.

In practice, it means you're still gonna be relying on setuptools.
(Presumably 0.7 post-merge, w/bdist_wheel support baked in.)  At some
point, there has to be a new way to do it, because the pain of
creating a functional dummy setup.py is a really high barrier to entry
for a build tool to meet, until all the current tools that run
setup.py files go away.

IMO it'd be better to standardize this bit *now*, so that it'd be
practical to start shipping projects without a setup.py, or perhaps
make a "one dummy setup.py to rule them all" implementation that
delegates everything to the new build interface.

I can certainly understand that there are more urgent priorities in
the short run; I just hope that a standard for this part lands
concurrent with, say, PEP 439 and distlib ending up in the stdlib, so
we don't have to wait another couple years to begin phasing out
setuptools/distutils as the only build game in town.

I mean, it basically amounts to defining some parameters to
programmatically call a pair of sdist() and bdist_wheel() functions
with, and a configuration syntax to say what distributions and modules
to import those functions from.  So it's not like it's going to be a
huge time drain.  (Maybe not even as much has already been consumed by
this thread so far.  ;-) )


Nick also wrote:
>> There's also an interesting migration problem for pre-2.0 sdists,
>> where we can't assume that "python setup.py bdist_wheel && pip install
>> <created wheel>" is equivalent to "python setup.py install": projects
>> like Twisted that run a post-install hook won't install properly if
>> you build a wheel first, since the existing post-install hook won't
>> run.
>>
>> It's an interesting problem, but one where my near term plans amount
>> to "document the status quo".

Yeah, it's already broken and the new world order isn't going to break
it any further.  Same goes for allowing pip to convert eggs; the ones
that don't work right due to bad platform tags, etc. *already* don't
work, so documenting the status quo as a transitional measure is
sufficient.  Heck, in general, supporting backward compatible stuff
that suffers from the same problems as the stuff it's being backward
compatible with is a no-brainer if it lets people get on the new so we
can phase out the old.

(Which is why I love that Vinay is looking into how to make wheels
more usable for some of eggs less-frequent but still important use
cases: it makes it that much easier to tell someone they don't need to
stay on setuptools to do the same stuff.)