[Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Glyph Lefkowitz]

> On Feb 16, 2016, at 3:05 AM, Matthias Klose <doko at ubuntu.com> wrote:
> 
> On 02.02.2016 02:35, Glyph Lefkowitz wrote:
>> 
>>> On Feb 1, 2016, at 3:37 PM, Matthias Klose <doko at ubuntu.com> wrote:
>>> 
>>> On 30.01.2016 00:29, Nathaniel Smith wrote:
>>>> Hi all,
>>>> 
>>>> I think this is ready for pronouncement now -- thanks to everyone for
>>>> all their feedback over the last few weeks!
>>> 
>>> I don't think so.  I am biased because I'm the maintainer for Python in Debian/Ubuntu.  So I would like to have some feedback from maintainers of Python in other Linux distributions (Nick, no, you're not one of these).
>> 
>> Possibly, but it would be very helpful for such maintainers to limit their critique to "in what scenarios will this fail for users" and not have the whole peanut gallery chiming in with "well on _my_ platform we would have done it _this_ way".
>> 
>> I respect what you've done for Debian and Ubuntu, Matthias, and I use the heck out of that work, but honestly this whole message just comes across as sour grapes that someone didn't pick a super-old Debian instead of a super-old Red Hat.  I don't think it's promoting any progress.
> 
> You may call this sour grapes, but in the light of people installing
> these wheels to replace/upgrade system installed eggs, it becomes an issue. It's fine to use such wheels in a virtual environment, however people tell users to use these wheels to replace system installed packages, distros will have a problem identifying issues.

I am 100% on board with telling people "don't use `sudo pip install´".  Frankly I have been telling the pip developers to just break this for years (see https://pip2014.com, which, much to my chagrin, still exists); `sudo pip install´ should just exit immediately with an error; to the extent that packagers need it, the only invocation that should work should be `sudo pip install --i-am-building-an-operating-system´.

But `sudo pip install´ of arbitrary packages is now, and always has been, basically broken; this PEP doesn't change that in any way I can see.  Specifically, since there are tools in place to ensure that the extension modules will load just fine, this won't be any more broken than `sudo pip install´-ing random C extension modules is today.  If anything it will be more reliable, since a lot of people already build and ship wheels to their production linux environments, and don't always understand the nuances around having to build on a system with a native package set that exactly matches their target environment.

> There is a substantial amount of extensions built using C++; I didn't check how many of these in c++0x/c++11 mode.  Until GCC 5, the c++11 ABI wasn't stable, and upstream never promised forward compatibility, something that even distros have to care about (usually by rebuilding packages before a release).  So if you want a lowest common denominator, then maybe limit or recommend the use of c++98 only.

Isn't this irrelevant as long as your entry-points are all 'extern "C"' and your C++ code statically links libstdc++?  The build toolchain in question doesn't include a dynamic libstdc++, does it?  If so, that's a pretty concrete problem with this proposal and it should be addressed.

>>> The proposal just takes some environment and declares that as a standard.  So everybody wanting to supply these wheels basically has to use this environment.
>> 
>> There's already been lots of discussion about how this environment is a lowest common denominator.  Many other similar environments could _also_ be lowest common denominator.
> 
> sure, but then please call it what it is. centos5 or somelinux1.

The point of the wheel tag is that its output should work on many linuxes.  A 'centos5' tag would imply that you can use arbitrary dynamic libraries (and perhaps even arbitrary packages!) from centos5, of which there are many; you can't, because auditwheel will yell at you.  It's the build environment plus restrictions around what you can depend on from that environment.

>> In the future, more specific and featureful distro tags sound like a good idea.  But could we please stop making the default position on distutils-sig "this doesn't cater to my one specific environment in the most optimal possible way, so let's give up on progress entirely"?  This is a good proposal that addresses environment portability and gives Python a substantially better build-artifact story than it currently has, in the environment most desperately needing one (server-side linux).  Could it be better?  Of course.  It could be lots better.  There are lots of use-cases for dynamically linked wheels and fancy new platform library features in newer linuxes.  But that can all come later, and none of it needs to have an impact on this specific proposal, right now.
> 
> I'm unsure how more specific and featureful distro tags will help, unless you start building more than one binary version of a wheel.

Yes, that would be the point of the tags.  When an ISV is targeting multiple platforms, they build multiple artifacts.  Distros are different platforms (with a common small subset platform, which we are calling 'manylinux' here) and so eventually being able to build things which target more than one of them would be a good thing.  But this is starting to get a bit wide of the real point...

> From a distro point of view I only can discourage using such wheels outside a virtual environment, and I would like to see a possibility to easily identify such wheels, something like loading a binary kernel module is tainting the kernel. This way distros can point users to the provider of such wheels.

Something like https://github.com/pypa/auditwheel you mean?

> This is not a "this doesn't cater to my one specific environment" position. Of course you probably get less push back from other distributions closer to the specific environment specified in the pep.  But please acknowledge that there might be issues that you and me don't yet even see, and provide a possibility to identify such issues.

The possibility to identify such issues would be to report bugs on https://github.com/pypa/auditwheel/issues and https://github.com/pypa/manylinux/issues and fix them.  Fundamentally I don't see anything wrong here; it is of course possible that it misses something, but if it misses something, it would be something *specific* (like the potential C++ issue you pointed about, which I'm not 100% sure about).

> At least Debian/Ubuntu took a long ride to avoid "accidental" interaction with local Python installations and local installations into the system installed Python.  For me this PEP feels like a step back, promising too much (manylinux), not pointing out possible issues, and giving no help in identifying possible issues with these wheels.

This whole section is about a tool to automatically identify possible issues with these wheels - https://www.python.org/dev/peps/pep-0513/#auditwheel - so I don't even really know what you mean by this comment.  I thought that the existence of this tool is one of the best parts of this PEP!

-glyph

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160216/9f3d7cc3/attachment.html>