[Distutils] draft PEP: manylinux1

[Nate Coraor]

Nathaniel, Robert, I'm really excited to see how quickly you're making
progress. A few comments below as I haven't had a chance to catch up on the
day's discussion:

On Wed, Jan 20, 2016 at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:

> Building on the compability lessons learned from these companies, we thus
> define a baseline ``manylinux1`` platform tag for use by binary Python
> wheels, and introduce the implementation of preliminary tools to aid in the
> construction of these ``manylinux1`` wheels.
>

Just a standards question: does this still require an update to PEP 425, or
would the definition of the  manylinux1 platform here supersede that
section of 425?

* Eventually, in the future, there may exist distributions that break
>   compatibility with this profile
>


> To handle the third case, we propose the creation of a file
> ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
> contents: ::
>
>    [manylinux1]
>    compatible = true
>
> where the supported values for the ``manylinux1.compatible`` entry are the
> same as those supported by the ConfigParser ``getboolean`` method.
>

Could this instead use the more powerful json-based syntax proposed by Nick
here:

    https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html

I have already implemented support for this in pip and wheel.


> Security Implications
> =====================
>
> One of the advantages of dependencies on centralized libraries in Linux is
> that bugfixes and security updates can be deployed system-wide, and
> applications which depend on on these libraries will automatically feel the
> effects of these patches when the underlying libraries are updated. This
> can
> be particularly important for security updates in packages communication
> across the network or cryptography.
>
> ``manylinux1`` wheels distributed through PyPI that bundle
> security-critical
> libraries like OpenSSL will thus assume responsibility for prompt updates
> in
> response disclosed vulnerabilities and patches. This closely parallels the
> security implications of the distribution of binary wheels on Windows that,
> because the platform lacks a system package manager, generally bundle their
> dependencies. In particular, because its lacks a stable ABI, OpenSSL
> cannot be
> included in the ``manylinux1`` profile.
>

I appreciate that this was addressed. I don't want to be responsible for
keeping the versions of these things up to date. So instead, I made a
docker-based build system that builds a ton of wheels on different
distros/versions using a common definition. But I appreciate that it's a
bit heavy and not everyone will prefer this.

One piece that is not yet complete in the work I've done so far is actually
ensuring that the external dependencies are installed, and providing some
feedback on what's missing. But that can be done.


> Rejected Alternatives
> =====================
>
> One alternative would be to provide separate platform tags for each Linux
> distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
> ``debian_jessie``, etc. Nothing in this proposal rules out the possibility
> of
> adding such platform tags in the future, or of further extensions to wheel
> metadata that would allow wheels to declare dependencies on external
> system-installed packages. However, such extensions would require
> substantially
> more work than this proposal, and still might not be appreciated by package
> developers who would prefer not to have to maintain multiple build
> environments
> and build multiple wheels in order to cover all the common Linux
> distributions.
> Therefore we consider such proposals to be out-of-scope for this PEP.
>

;)

For anyone who's interested, the next release of Galaxy (a popular
bioinformatics framework for running tools, workflows, etc.), due next
week, will ship with our modified pip that includes support for
distro/version-specific platform tags in wheels. All but one of our
dependent package's wheels are built with the generic `linux_x86_64` tag on
Debian Squeeze and will work with most distros, though, so we're basically
doing a "loose" version of manylinux1. Only our psycopg2 wheels are built
per-distro/version. I'm happy to see a more rigid definition for what we're
doing with the "generic" ones, this is certainly necessary should support
for generic Linux wheels ever be allowed into PyPI. manylinux1 and this PEP
seem to me like the right idea to do this.

We build these distro/version wheels using a modified wheel and the
aforementioned docker-based build system, called Starforge. Here's where
all of it lives:

Galaxy: https://github.com/galaxyproject/galaxy
Starforge: https://github.com/galaxyproject/starforge
"Galaxy pip": https://github.com/natefoo/pip/tree/linux-wheels
"Galaxy wheel": https://bitbucket.org/natefoo/wheel

I say all of this (again) because I think there's still a place for the
work we've done even if manylinux1 is accepted. Both manylinux1 *and* more
specific distro/version platform tags can happily coexist. So I want to
publicize the fact that this work is already done and in use (although I
know there are some things that would need to be done before it could be
upstreamed).

Basically I'm trying to determine whether there's any interest in this from
the pip and PyPI developers. If so, I'd be happy to write a PEP that
complements the one written by Robert and Nathaniel so that both types of
platform tags and wheels using them can be supported.

--nate
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/656c22f9/attachment-0001.html>