[Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 27 January 2016 at 16:37, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Jan 26, 2016 at 5:08 PM, Donald Stufft <donald at stufft.io> wrote:
>> As many know there has been an effort to get a generalized interface that
>> a build system can implement so that we can break the hard dependency on
>> setuptools that Python packaging currently has. After a lot of deliberation
>> and threads back and forth (as well as video meetings!) I think that the
>> version of the PEP that is currently on the PR in
>> https://github.com/pypa/interoperability-peps/pull/54 looks like it’s
>> generally the right thing to move forward with. I made a few little comments
>> but overall I think it’s there and we’re ready to move forward on trying to
>> get a reference implementation done that can validate the decisions made in
>> that PEP (and then, hopefully finalize the PEP and merge those
>> implementations).
>>
>> So many thanks to everyone involved in hammering this out thus far :) I’m
>> nervous but excited about the possibility of making setuptools just another
>> build system.
>
> AFAICR at the time the set of public threads petered out (I guess because
> Robert and I both got busy with other things), there were several
> fundamental disagreements still unresolved:

There were two pep drafts in flight, but I would hesitate to call it a
fundamental disagreement :).

> - You and Paul were strongly in favor of splitting up "working directories"
> and "sdists"; Robert and I didn't like the idea. The argument in favor is
> that we have to support working directory -> sdist and sdist -> wheel, so
> adding working directory -> wheel as an additional pathway creates larger
> test matrices and more opportunities for things to go wrong; the argument
> against is that this requires a new sdist format definition (currently it's
> basically "a zipped up working dir"), and it breaks like incremental
> rebuilds, which are a critical feature for developer adoption.

That was something that occurred in the rabbit-hole discussion of your
first draft; The PR Donald is proposing to accept is I think the
fourth major iteration? Two from you, two from me - building on
feedback each time. While I don't think Donalds position here has
changed, the draft in question doesn't alter any semantics of anything
- it captures a subset of the semantics such that the existing
behaviour of pip can be modelled on the resulting interface. In
particular, for this question, it retains 'develop', which is the
primary developer mode in use today: it has no implications for or
against incremental builds - it doesn't alter pips copy-before-build
behaviour, which pip already has for non-develop installs. E.g. its
moot; we can debate that further - and i'm sure we shall - but it has
no impact on the interface.

> - Robert was in favor of a command-line based interface; I favored a hook
> based interface. I won't try to summarize the disagreement here b/c I'm
> unable to articulate any advantages of the command-line based interface :-)

it has the fairly large advantage that the developers of pip are in
favour of it - as the folk that will have to deal with and debug what
goes on, thats more than sufficient for me.

> - Robert was in favor of preserving the current 'setup.py develop' semantics
> as-is, and in particular keeping the current behavior where it's the build
> system's job to pick which directory the .egg-link file should be installed
> into and then do the actual work; I favored an interface where the build
> system says "here's the path to a directory with the package and metadata",
> and then the installation tool (pip) picks the location and installs the
> .egg-link file (or whatever) pointing to that directory. The argument in
> favor of the current semantics is that hey, we need to get this show on the
> road, and this will require less code added to pip. The information is
> against is that the current design gets the separation of concerns wrong,
> impossible to implement without relying on unstandardized features like
> egg-link, and also I don't think it actually works right currently (e.g. if
> the user supplies --user or --root to the pip install -e command, then what
> happens? But I could be wrong -- haven't had a chance to fully reverse
> engineer pip/setuptools interaction here).

That feels like an exaggeration of my position. I'd love to evolve
those semantics. But since such an evolution is a hard API break
across everything, I don't want to make fixing the much more
significant problems of setup_requires and having to use setuptools
conditional on that other thing: its a massive adoption hurdle I
suspect.

> My personal feeling is that on the first two points my position is
> objectively correct ;-), while the third is more confusing and more of a
> judgement call.
>
> I haven't looked at the current version of the PEP, so the above may be out
> of date, but I think it's accurate WRT what's been discussed on the mailing
> list. Have there been some private video meetings where all these issues got
> hashed out somehow, or...?

No; there was the one video call, which was advertised here, and then
the subsequent drafts. The PR hasn't been changed in a couple of
months: in that time we've had Xmas, various work things, hacking
older pip version compat stuff into pbr, and me nagging Donald a lot
to get a review of the PR :).

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud