[Wheel-builders] libpythonX.Y.so.1

[Nate Coraor]

On Tue, Mar 1, 2016 at 4:58 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Mar 1, 2016 at 11:18 AM, Nate Coraor <nate at bx.psu.edu> wrote:
> > So I am working on building my first set of manylinux1 wheels. The first
> > candidate is psycopg2. I was able to generate the wheel which brings in
> all
> > the necessary libs:
> [...]
> > However, attempting to use this wheel on Ubuntu 14.04 results in a
> failure
> > to load libpython2.7.so.1:
> >
> >   ImportError: libpython2.7.so.1.0: cannot open shared object file: No
> such
> > file or directory
> >
> > PEP 513 addresses libpythonX.Y.so.1 but only to say that it does not
> need to
> > be linked, nor should it be. Is my understanding correct, then, that I
> would
> > need to fix psycopg2 myself to *not* link to libpython (if this works -
> > won't there be unresolved symbols at link time?)? Is there a more general
> > solution to this problem?
>
> Right, you need to not link to libpython. This does work -- it turns
> out that the linker doesn't check for unresolved symbols unless you
> specifically ask it to (via -Wl,--no-undefined or similar), I guess
> because the people who write linkers consider what we're doing here to
> be a valid use case :-).
>
> Are you using the docker images to build these? I thought we fixed it
> to not link to libpython by default, but if not then we should... (If
> your python is built with --disable-shared, which is the default when
> building from source, then by default any wheels you build with that
> python will not link to libpython. Unfortunately CentOS's python is
> built with --enable-shared. This is one of the reasons that the docker
> images build their own python instead of using the one shipped by
> CentOS.)
>

I'm using the docker image and the purpose-built Python, yeah. It's a
container that I started a few weeks ago though, so I can pull a new
version of the image and see if it's fixed. The Python in my version of the
image definitely has a shared libpython in the purpose-built Python build:

libpython2.7.so.1.0 => /opt/2.7.11/lib/libpython2.7.so.1.0
(0x00007fab9bdfd000)


>
> Also, I guess Robert is a bit distracted by thesis-ing, but I think he
> was at least planning to make auditwheel check for and reject wheels
> that link to libpython (or possibly repair them -- I think patchelf
> can do that). Perhaps he'll speak up about that, or I'm sure he'd
> welcome a pull request :-). I guess that's the more-general solution
> you're looking for.
>

A good idea, but if the docker image isn't supposed to produce wheels that
link libpython, that's also a good general solution for me. =)


>
> > Related, some of these are concerning to me. Although CentOS 5 receives
> > updates, so its OpenSSL should be secure and bug-free, this is still
> OpenSSL
> > 0.9.8e. And bundling SELinux and Kerberos/GSSAPI libs also makes me a bit
> > worried.
>
> I agree, it is worrying! But there's not much that can be done about
> it: if you want to make a single wheel that works across lots of linux
> distributions, and you need openssl, then you need to ship openssl,
> and that means that you need to take responsibility for keeping an eye
> out for security releases and respin your wheels when they happen.
>
> I guess the psycopg2 developers are already doing this for their
> Windows wheels, so perhaps they'll be willing to take on that
> responsibility for Linux wheels too? Or there might be things we can
> do to mitigate this somewhat: e.g., one can imagine the different
> folks who need to ship openssl getting together and building some
> shared infrastructure, so that they can share the burden -- like have
> one set of build scripts to update openssl and rebuild all their
> packages. Or once the native-libraries-in-wheels effort gets off the
> ground (sorry, I've been meaning to send out an email to kick that off
> but have been distracted by other things), then they'll be able to
> share a single openssl wheel and then that'll also centralize the
> update process. But fundamentally, the unavoidable fact is that if
> someone's not willing to take on responsibility for this somehow, then
> probably it's a bad idea for that person to ship manylinux wheels (or
> windows wheels, for that matter).
>

Sure, this makes sense to me. I wonder if it would be a good idea to have
something like Conda in the docker image that would make installing
up-to-date versions of these sorts of dependencies trivial.

Thanks,
--nate

P.S., the archives for this list aren't showing up on mail.python.org,
could you take a look in the list admin settings and see if they're enabled?



>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20160302/225a1a7c/attachment-0001.html>