[Distutils] Working toward Linux wheel support

[Nate Coraor]

On Tue, Sep 8, 2015 at 10:10 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft <donald at stufft.io> wrote:
> > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote:
> >> >>>
> >> >>> I'll create PRs for this against wheel and pip shortly. I can also
> work
> >> >>> on a PEP for the platform tag - I don't think it's going to need to
> be a
> >> >>> big one. Are there any preferences as to whether this should be a
> new PEP
> >> >>> or an update to 425?
> >> >>>
> >
> > Coming back to this, I'm wondering if we should include the libc
> > implementation/version in a less generic, but still generic linux wheel.
> Right
> > now if you staticly link I think the only platform ABIs you need to
> worry about
> > are libc and Python itself. Python itself is handled already but libc is
> not.
> > The only thing I've seen so far is "build on an old enough version of
> glibc
> > that it handles anything sane", but not all versions of Linux even use
> glibc at
> > all.
>
> This feels kinda half-baked to me?
>
> "linux" is a useful tag because it has a clear meaning: "there exists
> a linux system somewhere that can run this, but no guarantees about
> which one, good luck". When building a wheel it's easy to tell whether
> this tag can be correctly applied.
>

I'm not sure how it'd be possible to tell. The same meaning for a generic
tag would be true of any wheel built, regardless of whether the wheel has
dependencies in addition to libc.


> Distro-specific tags are useful because they also have a fairly clear
> meaning: "here's a specific class of systems that can run this, so
> long as you install enough packages to fulfill the external
> dependencies". Again, when building a wheel it's pretty easy to tell
> whether this tag can be correctly applied. (Of course someone could
> screw this up, e.g. by building on a system is technically distro X
> but has some incompatible hand-compiled libraries installed, but 99%
> of the time we can guess correctly.)
>
> If we define a LSB-style base system and give it a tag, like I don't
> know, the "Python base environment", call it "linux_pybe1_core" or
> something, that describes what libraries are and aren't available and
> their ABIs, and provide docs/tooling to help people explicitly create
> such wheels and check whether they're compatible with their system,
> then this is also useful -- we have proof that this is sufficient to
> actually distribute arbitrary software usefully, given that multiple
> distributors have converged on this strategy. (I've been talking to
> some people off-list about maybe actually putting together a proposal
> like this...)
>
> To me "linux_glibc_2.18" falls between the cracks though. If this
> starts being what you get by default when you build a wheel, then
> people will use it for wheels that are *not* statically linked, and
> what that tag will mean is "there exists some system that can run
> this, and that has glibc 2.18 on it, and also some other unspecified
> stuff, good luck". Which is pretty useless -- we might as well just
> stick with "linux" in this case. OTOH if it's something that builders
> have to opt into, then we could document that it's only to be used for
> wheels that are statically linked except for glibc, and make it mean
> "*any* system which has glibc 2.18 or later on it can run this". Which
> would be useful in some cases.


This is a tooling issue. If wheel (the package) inspects the built .so
files and finds they are not dynamically linked to anything not included
with glibc, it can apply the glibc tag. Otherwise, it'd apply the distro
tag. There's no possibility for human error here, unless they explicitly
override the platform tag.


> But at this point it's basically a
> version of the "defined base environment" approach, and once you've
> gone that far you might as well take advantage of the various
> distributors' experience about what should actually be in that
> environment -- glibc isn't enough.
>

While I agree that glibc isn't always enough, defining a base environment
that may not be met by the "standard" install of popular distributions
makes unprivileged wheel installation much more difficult. It's also not
going to work out of the box on older distributions that wouldn't provide
whatever standardized mechanism is defined for a list of "base environments
currently provided by this system" (unless pip does the work itself at
runtime to determine whether a base environment is met). Maybe an important
question: how many popular packages with C Extensions have dependencies in
addition to glibc?


>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150909/9275cc4b/attachment.html>