[Distutils] Finishing up PEP 517

[Nick Coghlan]

On 26 June 2017 at 22:36, Nathaniel Smith <njs at pobox.com> wrote:
> On Jun 25, 2017 12:46 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
> On 25 June 2017 at 17:41, Nathaniel Smith <njs at pobox.com> wrote:
>> Maybe you're right and there are exactly 2 front-end use cases and it
>> will turn out that the current PEP addresses them perfectly. I don't
>> have a crystal ball; I'm making an argument from ignorance.
>
> I'm not - we have two concrete potential consumers of the interface
> (pip and tox, aka "build to use" and "build to test"), and I'm
> designing the interface to cover their needs (i.e. out-of-tree wheel
> builds and actual sdists).
>
> If we discover other use cases later, we'll worry about them then (and
> the easy of doing so is the nicest benefit of defining this as a
> Python API), but the temptation to design in hyper-flexibility now
> falls under YAGNI (You Ain' Gonna Need It).
>
>
> My proposal also covers their needs AFAICT?

No, as you don't know in your proposal whether or not build_sdist can
fail until after you've already called it, and you always need to call
both `get_build_sdist_requires()` and `build_sdist()`, even if you
only care about doing an out-of-tree wheel build.

That categorically rules out two-pass introspection based
implementations that first interrogate the backend to find out which
hooks it supports, and then use that to decide their execution
strategy.

Tangent: it turns out the word order difference between
`get_build_wheel_requires` and `prepare_wheel_build_files` hurts my
brain, so I've changed the latter to `prepare_wheel_input_files`
below. That keeps the common prefix with `prepare_wheel_metadata`,
while avoiding the confusing word order reversal relative to
`build_wheel`.

That is, the current PEP allows a frontend to do the following when
given an arbitrary source tree rather than an sdist archive:

1. Look for `prepare_wheel_input_files()` on the backend. If you find it:

- call `get_build_wheel_requires()`
- install the additional dependencies
- call `prepare_wheel_input_files()`
- call `build_wheel()`

2. If you don't find the input preparation hook:

- call `get_build_sdist_requires()`
- call `get_build_wheel_requires()`
- install both sets of additional dependencies
- call `build_sdist()`
- unpack the sdist into the build directory
- call `build_wheel()`

Failure of any hook in any way (whether that's raising an exception or
returning something that isn't in line with the requirements of PEP
517) indicates a failed build.

To put it another way, the key reason we can be confident that PEP 517
is comprehensive as currently drafted is that we only have two target
artifact types (sdist & wheel), one source archiving strategy
(in-tree), and two binary build strategies (in-tree or out-of-tree).

That means source archiving is straightforward:

- call `get_build_sdist_requires()`
- install the additional dependencies
- call `build_sdist()`

In-tree binary builds are also straightforward:

- call `get_build_wheel_requires()`
- install the additional dependencies
- call `build_wheel()`

Out-of-tree binary builds via sdist combine the two:

- call `get_build_sdist_requires()`
- call `get_build_wheel_requires()`
- install both sets of additional dependencies
- call `build_sdist()`
- unpack the sdist into the build directory
- call `build_wheel()`

(Frontends would also be free to use two distinct environments, one to
create the sdist, and then a separate one to build the wheel)

However, that last strategy has an undesirable property: it means that
installing the sdist archiving dependencies becomes a constraint on
out-of-tree builds, which is annoying for both frontends and backends.
Adding direct support for out-of-tree builds resolves that problem.

Out-of-tree binary builds via a dedicated hook:

- call `get_build_wheel_requires()`
- install the additional dependencies
- call `prepare_wheel_input_files()`
- call `build_wheel()`

By contrast, if we only define a "try it and see if it works" approach
for build_sdist, and provide no other way to request an out-of-tree
build, then frontends *have* to do the following:

- call `get_build_sdist_requires()`
- call `get_build_wheel_requires()`
- install both sets of additional dependencies
- call `build_sdist()`
- if it returns `NotImplemented`... umm, give up, same as if it
failed? Fall back to the bad "copy everything" default that pip is
trying to get away from?
- unpack the sdist into the build directory
- call `build_wheel()`

Without a dedicated out-of-tree build preparation hook, there's no way
for frontends to know in advance that build_sdist might fail, there's
no way for them to avoid installing the sdist archiving dependencies
when doing an out-of-tree build, and there's no way for backends to
provide a fallback build directory preparation strategy that relies
solely on the wheel building dependencies.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia