[Distutils] PEP: Build system abstraction for pip/conda etc

[Robert Collins]

On 27 January 2016 at 22:24, Paul Moore <p.f.moore at gmail.com> wrote:
> On 27 January 2016 at 05:39, Robert Collins <robertc at robertcollins.net> wrote:
>>> - You and Paul were strongly in favor of splitting up "working directories"
>>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>>> that we have to support working directory -> sdist and sdist -> wheel, so
>>> adding working directory -> wheel as an additional pathway creates larger
>>> test matrices and more opportunities for things to go wrong; the argument
>>> against is that this requires a new sdist format definition (currently it's
>>> basically "a zipped up working dir"), and it breaks like incremental
>>> rebuilds, which are a critical feature for developer adoption.
>>
>> That was something that occurred in the rabbit-hole discussion of your
>> first draft; The PR Donald is proposing to accept is I think the
>> fourth major iteration? Two from you, two from me - building on
>> feedback each time. While I don't think Donalds position here has
>> changed, the draft in question doesn't alter any semantics of anything
>> - it captures a subset of the semantics such that the existing
>> behaviour of pip can be modelled on the resulting interface. In
>> particular, for this question, it retains 'develop', which is the
>> primary developer mode in use today: it has no implications for or
>> against incremental builds - it doesn't alter pips copy-before-build
>> behaviour, which pip already has for non-develop installs. E.g. its
>> moot; we can debate that further - and i'm sure we shall - but it has
>> no impact on the interface.
>
> I'm still not clear on what the position is here. The PEP as I read it
> still offers no means to ask the build system to produce a sdist, so
> I'm concerned as to what will happen here.


> In the absence of a "pip sdist" command I can see that you're saying
> that we can implement pip's functionality without caring about this.
> But fundamentally, people upload sdists and wheels to PyPI. A build
> system that doesn't support sdists (which this PEP allows) will
> therefore have to publish wheel-only builds to PyPI, and I am very
> much against the idea of PyPI hosting "binary only" distributions.

As you know - https://mail.python.org/pipermail/distutils-sig/2015-March/026013.html
- flit /already/ does this. If we want to require sdists as a thing,
PyPI needs to change to do that. I don't care whether it does or
doesn't myself: folk that want to not ship sources will always find a
way. I predict empty sdists, for instance.

> If project foo has no sdist, "pip wheel foo" for a platform where
> there's no compatible wheel available will fail, as there's no way for
> pip to locate the source.

Thats true; but pip isn't a build system itself - and so far pip has
no role in the uploading of wheels to PyPI. Nor does - last I checked
- twine build things for you. The pattern is that developers prepare
there artifacts, then ask twine to upload them.

> So can we please revisit the question of whether build systems will be
> permitted to refuse to generate sdists? Note that I don't care whether
> we formally define a new sdist format, or go with something adhoc, or
> whatever. All I care about is that the PEP states that build systems
> must support generating a file that can be uploaded to PyPI and used
> by pip to build a wheel as described above (not "git clone this
> location and do 'pip wheel .'"). I think that making build systems
> expose how you make such a file by requiring a "sdist" subcommand is a
> reasonable approach, but I'm OK with naming the subcommand
> differently.

I truely have no opinion here. I don't think it harms the abstract
build system to have it, but I do know that Donald very much does not
want pip to have to know or care about building sdists per se, and he
may see this as an attractive nuisance.

Who / what tool do we expect to use the sdist command in the abstract interface?

> Thanks,
> Paul
>
> PS I agree with Nathaniel, insofar as there were definitely a number
> of unresolved points remaining after the various public discussions,
> and I don't think it's reasonable to say that any proposal is "ready
> for implementation" without a public confirmation that those issues
> have been resolved.

I suspect it had been quescient too long and folk had paged out the
state. Certainly I don't think Donald intended to bypass any community
debate, and neither did I.

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud