[Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 27 January 2016 at 20:30, Nathaniel Smith <njs at pobox.com> wrote:

>>> - Robert was in favor of a command-line based interface; I favored a hook
>>> based interface. I won't try to summarize the disagreement here b/c I'm
>>> unable to articulate any advantages of the command-line based interface :-)
>>
>> it has the fairly large advantage that the developers of pip are in
>> favour of it - as the folk that will have to deal with and debug what
>> goes on, thats more than sufficient for me.
>
> Can you link to where they said this?

Donald specifically:

https://mail.python.org/pipermail/distutils-sig/2015-November/027727.html
https://mail.python.org/pipermail/distutils-sig/2015-November/027722.html


>>> - Robert was in favor of preserving the current 'setup.py develop' semantics
>>> as-is, and in particular keeping the current behavior where it's the build
>>> system's job to pick which directory the .egg-link file should be installed
>>> into and then do the actual work; I favored an interface where the build
>>> system says "here's the path to a directory with the package and metadata",
>>> and then the installation tool (pip) picks the location and installs the
>>> .egg-link file (or whatever) pointing to that directory. The argument in
>>> favor of the current semantics is that hey, we need to get this show on the
>>> road, and this will require less code added to pip. The information is
>>> against is that the current design gets the separation of concerns wrong,
>>> impossible to implement without relying on unstandardized features like
>>> egg-link, and also I don't think it actually works right currently (e.g. if
>>> the user supplies --user or --root to the pip install -e command, then what
>>> happens? But I could be wrong -- haven't had a chance to fully reverse
>>> engineer pip/setuptools interaction here).
>>
>> That feels like an exaggeration of my position. I'd love to evolve
>> those semantics. But since such an evolution is a hard API break
>> across everything, I don't want to make fixing the much more
>> significant problems of setup_requires and having to use setuptools
>> conditional on that other thing: its a massive adoption hurdle I
>> suspect.
>
> Are you concerned that this would make it impossible to do a simple
> shim to run old setuptools projects under the new interface? I think
> this part is fine: basically for the hook that I propose, a
> PEPXX->setup.py shim would implement it by running
>
>   setup.py build_ext -i && setup.py egg_info


> which sets up the current directory to be (a) importable, and (b) have
> metadata, and then pip would be responsible for installing a .egg-link
> pointing to this directory. IIUC this is exactly what 'develop'
> currently does, except that it creates the .egg-link itself.

And it needs to do command line entry point processing, not that that
is super hard.

My concern is mainly about unknown unknowns. We've been blocked on
lots of incremental improvement because we've been taking really big
bites. I firmly believe we should take small bites: do one thing, do
it well. E.g. PEP 508, which really did just one thing - fixed the
python 2.7.10 < 2.7.9 issue - but even it took weeks of drafting,
testing, and the patches for it are still wending through review into
setuptools, pip and packaging, let along projects further out on the
periphery like pbr.


The one thing I want to accomplish here is enabling alternative build
systems, because it seem(ed) like there was lots of pent up demand for
that; I specifically don't want to reengineer what pip needs from the
build systems, nor what they need to deliver *in this iteration*: the
tech debt we have here is enormous: there is /no/ standard that we can
point at and say 'implement that', because its literally 'whatever pip
calls setup.py with', and that is subject to change whenever someone
puts a clever patch into pip.

My view is that we won't get it right, no matter what. We have so many
things we're still working on getting agreement on in the long view -
like the copy-before-build question; should progress bars be a thing?
What about incremental builds? What about Wheel format 2? ... once we
have a standard, that standard can be evolved both compatibly, and
incompatibly, with a reference baseline we can compare too, rather
than a defacto thing. We probably want to batch up the changes, and
not dribble them out once every few months - but I definitely see more
iterations happening.

> Doh, and of course that was the other substantive disagreement:
> whether to mandate egg_info.

Yeah - note that my PR punts on this - it says PEP 427 METADATA file,
but as we know the contents of that are a snapshot of earlier drafts
of PEP-426, so its really not all that well defined, and before folk
can iterate on that further we'll need to do something about that -
either by a rewrite of PEP 426 into a much more pithy thing now,
without the enterprise feel and the moon-shot aspects, or by issuing a
PEP that specifies exactly what is in METADATA.

> Well, at least now it looks like we can forget about that whole debate
> about the need to quickly get metadata from scipy sdists, since
> apparently scipy will have wheels on all supported platforms before
> pip implements either the new build system interface or the resolver
> :-). Of course there will still be an issue for more obscure platforms
> and for projects whose developers can't be arsed to provide wheels.

There is no install command in the interface; if projects can't
provide wheels, they can't migrate to the new abstract system - it may
be that some projects need an updated wheel spec to be able to
migrate, and it may also be that pip retains the ability to run
setup.py install for unmigrated projects forever (since pypi has deep
history), but from a design perspective, we are explicitly assuming
'can make wheels'. - Thats reasonable. 'making wheels is cheap' is not
so reasonable :).

> [...]
>> No; there was the one video call, which was advertised here, and then
>> the subsequent drafts. The PR hasn't been changed in a couple of
>> months: in that time we've had Xmas, various work things, hacking
>> older pip version compat stuff into pbr, and me nagging Donald a lot
>> to get a review of the PR :).
>
> Plus all your great work on getting PEP 508 sorted out so it wouldn't
> block this build system stuff :-).

Yah - that was a group effort though, I was just the ring leader :0

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud