[Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 16:22, Nathaniel Smith <njs at pobox.com> wrote:
...
> I'm sympathetic to the general approach, but on net I think I prefer a
> slightly different proposal.
>
> Downsides to just standardizing [setup_requires]:

If I write a PEP, it won't be standardising setup_requires, it will be
standardising bootstrap requirements. The distinction is nuance but
critical:

 - we don't expect setuptools to ever need to honour it (of course, it
could, should it choose to, and ease of adoption may suggest that it
should)
 - as a new feature, making it opt in allows folk to adopt it when
they are ready; if it was literally just a new home for
setup_requires, we may see build systems auto-populating it, and the
potential swamp you describe below would then apply.

> - if projects have existing ones, that's actually kinda good / kinda
> bad -- pip has never been respecting these before, so if we suddenly
> start doing that then existing stuff could break. I don't know how
> likely this is, but experience suggests that *something* will break
> and make someone angry. (I'm still blinking at getting angry
> complaints arguing that uploading a wheel to pypi, where before there
> was only an sdist, should be treated as a compatibility-breaking
> change that requires a new version etc.)

Yes, things will break: anyone using this will need a new pip, by
definition. Not everyone will be willing to wait 10 years before using
it :).

> - IMO an extremely valuable aspect of this new declarative
> setup-requirements thing is that it gives us an opportunity to switch
> to enforcing the accuracy of this metadata. Right now we have a swamp
> we need to drain, where there's really no way to know what environment
> any given setup.py needs to run. Sometimes there are setup_requires,
> sometimes not; if there are setup_requires then sometimes they're

Huh? I've not encountered any of this, ever. I'd love some examples to
go look at. The only issue I've ever had with setup_requires is the
easy_install stuff it ties into.
...
> instead of punishing them -- is if we make the rule be "projects that
> use the declarative setup-requirements feature also get isolated build
> environments". (Then the messaging is: "see, this helps you check that
> you actually set it up right! if your new metadata works for you in
> testing, it'll also work for your users!) But this again would mean we
> can't reuse the existing [setup_requires] section.

I'm very much against forcing isolated build environments as part of
this effort. I get where you are coming from, but really it conflates
two entirely separate things, and you'll *utterly* break building
anything with dependencies that are e.g. SWIG based unless you
increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
think).

Working around that will require a bunch of UX work, and its
transitive: you have to expose
how-to-workaround-the-fact-that-all-our-deps-are-not-installable all
the way up the chain. That's probably a good thing to do (e.g. see
bindep, or the aborted callout-to-system-packages we proposed after
PyCon AU last year), but tying these two things together is not
necessary, though I can certainly see the appeal; the main impact I
see is that it will just impede adoption.

The reality, AFAICT, is that most projects with undeclared build deps
today get corrected fairly quickly: a bug is filed, folk fix it, and
we move on. A robotic system that isolates everything such that folk
*cannot* fix it is much less usable, and I'm very much in favour of
pragmatism here.

> - setup.cfg is kind of a terrible format for standardizing things
> because the only definition of the format is "read the ConfigParser
> source". You cannot parse setup.cfg from a non-Python language. And
> the *only* benefit is that it already exists; teaching pip to read
> pybuild.json or pybuild.toml instead would be completely trivial. So
> if we aren't going to try and grandfather in existing [setup_requires]
> sections, then we might as well switch to a better file format at the
> same time -- this is not the hard part.

The patch to read a single list-valued key out of setup.cfg is trivial
and shallow. We've not managed to settle on consensus on a file format
choice in a year of debate. I hold zero confidence we will going
forward either. If the BDFL delegate makes a call - fine. I read
Nick's earlier email in the thread as such a thing TBH :).

> So I like the idea of splitting out the declarative setup-requirements
> PEP from the new build system hook interface PEP, but I think that the
> way we should do this is by defining a new pybuild.whatever file like
> the ones in PEP 516 / PEP 517 (they're identical in this regard) that
> *only* has schema-version and bootstrap-requirements, and then we can
> add the build backend key as a second step.

I think the risk there is that that presumes the answer, and without
the abstract build effort actually moving forward we may be years
before we actually see the file being used for anything else. If ever.

-Rob