[Distutils] PEP 517: Open questions around artifact export directories

[Nathaniel Smith]

On Wed, Jun 14, 2017 at 1:49 PM, Donald Stufft <donald at stufft.io> wrote:

>
>
> On Jun 12, 2017, at 9:27 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>
> 5) Drop the prepare_files hook
> Rationale: it's purpose is somewhat unclear at this point, and it
> gives up the main advantage of going through sdist (the guarantee that
> building from sdist and building from unpacked tree give the same
> results) while still having the main disadvantages (you have to copy
> everything and lose incremental rebuilds).
>
>
> The main reason this made it into the PEP is that build_sdist may have
> additional dependencies that prepare_build_files doesn't (e.g. Thomas
> has indicated that flit needs to call out to VCS tools to build a full
> sdist, but can export its own input files directly).
>
> So while it's technically a "make it faster" hook rather than an
> essential "make it right" hook, I think just smoothing out the
> interaction between pip & flit specifically provides sufficient value
> to make it worth including in the initial version of the
> specification.
>
>
> Just as an anecdote, today requests pushed a brown paper bag release
> because they created a wheel out of a checkout of the requests repository,
> which happened to have some .pyc files and directories left over from git
> and some refactoring. This caused the library to be completely broken as
> released in the wheel. This is the kind of error that creating a sdist
> ideally hopes to eliminate (and hopefully the make it faster hook doesn’t
> end up invalidating that point).
>

To clarify my position:

- My grumpiness last week was at the discussion around PEP 517 moving
towards assuming that tree->sdist->wheel is the *only* path that matters.
That's not something we can mandate in theory or in practice.

- If pip specifically wants to insist on generating sdists every time it
builds from a checkout, then I'm dubious but I've made my case and I can
live with the pip maintainers' decision.

- However, I am more dubious about hacking stuff like prepare_build_files
into the standard to support this specific interaction between two specific
tools, especially since it's not at all obvious to me that this is the best
way to support even those two tools. Our two concrete examples are:

1. distutils/setuptools, which clearly aren't to be trusted (the requests
example is great. I mean, it's wince-inducing, but it's a great and
convincing example why the gimme-release-artifacts command needs to go
through sdist). If you're worried about getting a reliable build here, the
solution is to build an sdist; prepare_build_files is irrelevant.

2. flit, which is totally trustworthy (in this regard), and for whom the
prepare_build_files *also* accomplishes nothing useful, since it will use
exactly the same logic to copy files from the tree->fake unpacked
sdist->wheel as it would if going from tree->wheel directly; there's
literally no difference. It's a bit cargo-culty isn't it? It makes flit's
install path look more like the distutils/setuptools one, but either we
trust flit to produce consistent results whether building from a tree or
sdist and it's unnecessary, or else we don't trust flit to produce
consistent results and then it's harmful and we should insist on a proper
sdist. What problem is this solving?

I don't really feel like we've fully explored this design space either. Can
flit really not make an sdist from an sdist? Or if it really can't, then we
should have some way to express this in the spec (maybe build_sdist signals
this by raising NotImplementedError?), and then pip needs to have some
strategy for dealing with this. If pip finds itself trying to build
something where the build backend says it can't make an sdist and can't do
prepare_build_files, then it needs to fall back on something, which I guess
is either shutil.copytree or doing an in-place build. Either seems about as
good as prepare_build_files to me, and they don't require risky
standardization work.

prepare_build_files is by far the most speculative part of the standard,
because it's trying to solve on paper problems we haven't yet solved in
code. That makes me nervous. It doesn't make me *as* nervous as if this
were a mandatory part of the spec, because if it turns out in a few years
that no-one actually uses it, then we can drop it from both frontends and
backends and just sort of pretend it never existed. But it makes me
somewhat nervous.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org <http://vorpus.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170615/d90b66a4/attachment.html>