[Distutils] moving things forward (was: wheel including files it shouldn't)

[Nathaniel Smith]

On Wed, May 4, 2016 at 10:42 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On 5 May 2016 at 16:22, Nathaniel Smith <njs at pobox.com> wrote:
> ...
>> I'm sympathetic to the general approach, but on net I think I prefer a
>> slightly different proposal.
>>
>> Downsides to just standardizing [setup_requires]:
>
> If I write a PEP, it won't be standardising setup_requires, it will be
> standardising bootstrap requirements. The distinction is nuance but
> critical:
>
>  - we don't expect setuptools to ever need to honour it (of course, it
> could, should it choose to, and ease of adoption may suggest that it
> should)
>  - as a new feature, making it opt in allows folk to adopt it when
> they are ready; if it was literally just a new home for
> setup_requires, we may see build systems auto-populating it, and the
> potential swamp you describe below would then apply.

The main argument I was making there was that it needs to be a new
opt-in thing, so if we're agreed on that then great :-).

>> - if projects have existing ones, that's actually kinda good / kinda
>> bad -- pip has never been respecting these before, so if we suddenly
>> start doing that then existing stuff could break. I don't know how
>> likely this is, but experience suggests that *something* will break
>> and make someone angry. (I'm still blinking at getting angry
>> complaints arguing that uploading a wheel to pypi, where before there
>> was only an sdist, should be treated as a compatibility-breaking
>> change that requires a new version etc.)
>
> Yes, things will break: anyone using this will need a new pip, by
> definition. Not everyone will be willing to wait 10 years before using
> it :).

Just to clarify (since we seem to agree): I meant that if pip starts
interpreting an existing setup.cfg thing, then the new-pip/old-package
situation could break, which would be bad.

>> - IMO an extremely valuable aspect of this new declarative
>> setup-requirements thing is that it gives us an opportunity to switch
>> to enforcing the accuracy of this metadata. Right now we have a swamp
>> we need to drain, where there's really no way to know what environment
>> any given setup.py needs to run. Sometimes there are setup_requires,
>> sometimes not; if there are setup_requires then sometimes they're
>
> Huh? I've not encountered any of this, ever. I'd love some examples to
> go look at. The only issue I've ever had with setup_requires is the
> easy_install stuff it ties into.

I don't think I've ever seen a package that had accurate
setup_requires (outside the trivial case of packages where
setup_requires=[] is accurate). Scientific packages in particular
universally have undeclared setup requirements.

> ...
>> instead of punishing them -- is if we make the rule be "projects that
>> use the declarative setup-requirements feature also get isolated build
>> environments". (Then the messaging is: "see, this helps you check that
>> you actually set it up right! if your new metadata works for you in
>> testing, it'll also work for your users!) But this again would mean we
>> can't reuse the existing [setup_requires] section.
>
> I'm very much against forcing isolated build environments as part of
> this effort. I get where you are coming from, but really it conflates
> two entirely separate things, and you'll *utterly* break building
> anything with dependencies that are e.g. SWIG based unless you
> increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
> think).

Okay, now's my turn to be baffled :-). I literally have no idea what
you're talking about here. What would this 10x longer PEP be talking
about? Why would this be needed?

> Working around that will require a bunch of UX work, and its
> transitive: you have to expose
> how-to-workaround-the-fact-that-all-our-deps-are-not-installable all
> the way up the chain. That's probably a good thing to do (e.g. see
> bindep, or the aborted callout-to-system-packages we proposed after
> PyCon AU last year), but tying these two things together is not
> necessary, though I can certainly see the appeal; the main impact I
> see is that it will just impede adoption.

What are these things that aren't pip-installable and why isn't the
solution to fix that? I definitely don't want to break things that
work now, but providing new features that incentivize folks to clean
up their stuff is a good thing, surely? Yeah, it means that the
bootstrap-requirements stuff will take some time and cleanup to
spread, but that's life.

We've spent a huge amount of effort on reaching the point where pretty
much everything *can* be made pip installable. Heck, *PyQt5*, which is
my personal benchmark for a probably-totally-unpackageable package,
announced last week that they now have binary wheels on pypi for all
of Win/Mac/Linux:

  https://pypi.python.org/pypi/PyQt5/5.6

I want to work towards a world where this stuff just works, not keep
holding the entire ecosystem back with compromise hacks to work around
a minority of broken packages.

> The reality, AFAICT, is that most projects with undeclared build deps
> today get corrected fairly quickly: a bug is filed, folk fix it, and
> we move on. A robotic system that isolates everything such that folk
> *cannot* fix it is much less usable, and I'm very much in favour of
> pragmatism here.

Again, in my world ~100% of packages have undeclared build deps...

>> - setup.cfg is kind of a terrible format for standardizing things
>> because the only definition of the format is "read the ConfigParser
>> source". You cannot parse setup.cfg from a non-Python language. And
>> the *only* benefit is that it already exists; teaching pip to read
>> pybuild.json or pybuild.toml instead would be completely trivial. So
>> if we aren't going to try and grandfather in existing [setup_requires]
>> sections, then we might as well switch to a better file format at the
>> same time -- this is not the hard part.
>
> The patch to read a single list-valued key out of setup.cfg is trivial
> and shallow. We've not managed to settle on consensus on a file format
> choice in a year of debate. I hold zero confidence we will going
> forward either. If the BDFL delegate makes a call - fine. I read
> Nick's earlier email in the thread as such a thing TBH :).

Oh sure, I think everyone agrees that the file format choice is not a
make-or-break decision and that ultimately this is a good case for of
BDFL-style pronouncement rather than endless consensus seeking. But
I'm still allowed to make technical arguments for why I think the
BDFL-delegate should pronounce one way or another :-).

>> So I like the idea of splitting out the declarative setup-requirements
>> PEP from the new build system hook interface PEP, but I think that the
>> way we should do this is by defining a new pybuild.whatever file like
>> the ones in PEP 516 / PEP 517 (they're identical in this regard) that
>> *only* has schema-version and bootstrap-requirements, and then we can
>> add the build backend key as a second step.
>
> I think the risk there is that that presumes the answer, and without
> the abstract build effort actually moving forward we may be years
> before we actually see the file being used for anything else. If ever.

I see the risk, but I think it's minimal. We're not presuming anything
about the answer except that somehow it will involve adding some kind
of static metadata, which seems very safe.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org