[Distutils] moving things forward (was: wheel including files it shouldn't)

[Robert Collins]

On 5 May 2016 at 18:32, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, May 4, 2016 at 10:42 PM, Robert Collins
>...
>> Yes, things will break: anyone using this will need a new pip, by
>> definition. Not everyone will be willing to wait 10 years before using
>> it :).
>
> Just to clarify (since we seem to agree): I meant that if pip starts
> interpreting an existing setup.cfg thing, then the new-pip/old-package
> situation could break, which would be bad.

No. Old pip new package will break, new pip old package is entirely safe AFAICT.

>>> - IMO an extremely valuable aspect of this new declarative
>>> setup-requirements thing is that it gives us an opportunity to switch
>>> to enforcing the accuracy of this metadata. Right now we have a swamp
>>> we need to drain, where there's really no way to know what environment
>>> any given setup.py needs to run. Sometimes there are setup_requires,
>>> sometimes not; if there are setup_requires then sometimes they're
>>
>> Huh? I've not encountered any of this, ever. I'd love some examples to
>> go look at. The only issue I've ever had with setup_requires is the
>> easy_install stuff it ties into.
>
> I don't think I've ever seen a package that had accurate
> setup_requires (outside the trivial case of packages where
> setup_requires=[] is accurate). Scientific packages in particular
> universally have undeclared setup requirements.

Are those requirements pip installable?
..
>> I'm very much against forcing isolated build environments as part of
>> this effort. I get where you are coming from, but really it conflates
>> two entirely separate things, and you'll *utterly* break building
>> anything with dependencies that are e.g. SWIG based unless you
>> increase the size of the PEP by about 10-fold. (Thats not hyperbole, I
>> think).
>
> Okay, now's my turn to be baffled :-). I literally have no idea what
> you're talking about here. What would this 10x longer PEP be talking
> about? Why would this be needed?

Take an i386 linux machine, and build something needing pyqt5 on it
:). Currently, you apt-get/yum/dnf/etc install python-pyqt5, then run
pip install. If using a virtualenv you enable system site packages.

When you introduce isolation, the build will only have the standard
library + whatever is declared as a dep: and pyqt5 has no source on
PyPI.

So the 10x thing is defining how the thing doing the isolation (e.g.
pip) should handle things that can't be installed but are already
available on the system.

And that has to tunnel all the way out to the user, because its
context specific, its not an attribute of the dependencies per se
(since new releases can add or remove this situation), nor of the
consuming thing (same reason).

Ultimately, its not even an interopability question: pip could do
isolated builds now, if it chose, and it has no ramifications as far
as PEPs etc are concerned.
...
> What are these things that aren't pip-installable and why isn't the
> solution to fix that? I definitely don't want to break things that
> work now, but providing new features that incentivize folks to clean
> up their stuff is a good thing, surely? Yeah, it means that the
> bootstrap-requirements stuff will take some time and cleanup to
> spread, but that's life.

We've a history in this group of biting off too much and things not
getting executed. We're *still* in the final phases of deploying
PEP-508, and it was conceptually trivial. I'm not arguing that we
shouldn't make things better, I'm arguing that tying two separate
things together because we *can* seems, based on the historical
record, to be unwise.

> We've spent a huge amount of effort on reaching the point where pretty
> much everything *can* be made pip installable. Heck, *PyQt5*, which is
> my personal benchmark for a probably-totally-unpackageable package,
> announced last week that they now have binary wheels on pypi for all
> of Win/Mac/Linux:
>
>   https://pypi.python.org/pypi/PyQt5/5.6
>
> I want to work towards a world where this stuff just works, not keep
> holding the entire ecosystem back with compromise hacks to work around
> a minority of broken packages.

Sure, but the underlying problem here is that manylinux is a 90%
solve: its great for the common cases but it doesn't actually solve
the actual baseline problem: we can't represent the actual system
dependencies needed to rebuild many Python packages.

pyqt5 not having i386 is just a trivial egregious case. ARM32 and 64
is going to be super common, Power8 another one, let alone less common
but still extant and used architectures like PPC, itanium, or new ones
like x86_32 [If I remember the abbreviation right - no, its not i386].

Solve that underlying problem - great, then isolation becomes an
optimisation question for things without manylinux wheels. But if we
don't solve it then isolation becomes a 'Can build X at all' question,
which is qualitatively different.

I'm all for solving the underlying problem, but not at the cost of
*not solving* the 'any easy-install is triggered' problem for another
X months while that work takes place.


>> The reality, AFAICT, is that most projects with undeclared build deps
>> today get corrected fairly quickly: a bug is filed, folk fix it, and
>> we move on. A robotic system that isolates everything such that folk
>> *cannot* fix it is much less usable, and I'm very much in favour of
>> pragmatism here.
>
> Again, in my world ~100% of packages have undeclared build deps...

So - put a patch forward to pip to do isolated builds. If/when bug
reports come in, we can solve them there. There's no standards body
work involved in that as far as I can see....

-Rob


-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud