[Distutils] PEP: Build system abstraction for pip/conda etc

[Paul Moore]

On 27 January 2016 at 05:39, Robert Collins <robertc at robertcollins.net> wrote:
>> - You and Paul were strongly in favor of splitting up "working directories"
>> and "sdists"; Robert and I didn't like the idea. The argument in favor is
>> that we have to support working directory -> sdist and sdist -> wheel, so
>> adding working directory -> wheel as an additional pathway creates larger
>> test matrices and more opportunities for things to go wrong; the argument
>> against is that this requires a new sdist format definition (currently it's
>> basically "a zipped up working dir"), and it breaks like incremental
>> rebuilds, which are a critical feature for developer adoption.
>
> That was something that occurred in the rabbit-hole discussion of your
> first draft; The PR Donald is proposing to accept is I think the
> fourth major iteration? Two from you, two from me - building on
> feedback each time. While I don't think Donalds position here has
> changed, the draft in question doesn't alter any semantics of anything
> - it captures a subset of the semantics such that the existing
> behaviour of pip can be modelled on the resulting interface. In
> particular, for this question, it retains 'develop', which is the
> primary developer mode in use today: it has no implications for or
> against incremental builds - it doesn't alter pips copy-before-build
> behaviour, which pip already has for non-develop installs. E.g. its
> moot; we can debate that further - and i'm sure we shall - but it has
> no impact on the interface.

I'm still not clear on what the position is here. The PEP as I read it
still offers no means to ask the build system to produce a sdist, so
I'm concerned as to what will happen here.

In the absence of a "pip sdist" command I can see that you're saying
that we can implement pip's functionality without caring about this.
But fundamentally, people upload sdists and wheels to PyPI. A build
system that doesn't support sdists (which this PEP allows) will
therefore have to publish wheel-only builds to PyPI, and I am very
much against the idea of PyPI hosting "binary only" distributions.

If project foo has no sdist, "pip wheel foo" for a platform where
there's no compatible wheel available will fail, as there's no way for
pip to locate the source.

So can we please revisit the question of whether build systems will be
permitted to refuse to generate sdists? Note that I don't care whether
we formally define a new sdist format, or go with something adhoc, or
whatever. All I care about is that the PEP states that build systems
must support generating a file that can be uploaded to PyPI and used
by pip to build a wheel as described above (not "git clone this
location and do 'pip wheel .'"). I think that making build systems
expose how you make such a file by requiring a "sdist" subcommand is a
reasonable approach, but I'm OK with naming the subcommand
differently.

Thanks,
Paul

PS I agree with Nathaniel, insofar as there were definitely a number
of unresolved points remaining after the various public discussions,
and I don't think it's reasonable to say that any proposal is "ready
for implementation" without a public confirmation that those issues
have been resolved.