[Wheel-builders] libpythonX.Y.so.1

[Nathaniel Smith]

On Tue, Mar 1, 2016 at 11:18 AM, Nate Coraor <nate at bx.psu.edu> wrote:
> So I am working on building my first set of manylinux1 wheels. The first
> candidate is psycopg2. I was able to generate the wheel which brings in all
> the necessary libs:
[...]
> However, attempting to use this wheel on Ubuntu 14.04 results in a failure
> to load libpython2.7.so.1:
>
>   ImportError: libpython2.7.so.1.0: cannot open shared object file: No such
> file or directory
>
> PEP 513 addresses libpythonX.Y.so.1 but only to say that it does not need to
> be linked, nor should it be. Is my understanding correct, then, that I would
> need to fix psycopg2 myself to *not* link to libpython (if this works -
> won't there be unresolved symbols at link time?)? Is there a more general
> solution to this problem?

Right, you need to not link to libpython. This does work -- it turns
out that the linker doesn't check for unresolved symbols unless you
specifically ask it to (via -Wl,--no-undefined or similar), I guess
because the people who write linkers consider what we're doing here to
be a valid use case :-).

Are you using the docker images to build these? I thought we fixed it
to not link to libpython by default, but if not then we should... (If
your python is built with --disable-shared, which is the default when
building from source, then by default any wheels you build with that
python will not link to libpython. Unfortunately CentOS's python is
built with --enable-shared. This is one of the reasons that the docker
images build their own python instead of using the one shipped by
CentOS.)

Also, I guess Robert is a bit distracted by thesis-ing, but I think he
was at least planning to make auditwheel check for and reject wheels
that link to libpython (or possibly repair them -- I think patchelf
can do that). Perhaps he'll speak up about that, or I'm sure he'd
welcome a pull request :-). I guess that's the more-general solution
you're looking for.

> Related, some of these are concerning to me. Although CentOS 5 receives
> updates, so its OpenSSL should be secure and bug-free, this is still OpenSSL
> 0.9.8e. And bundling SELinux and Kerberos/GSSAPI libs also makes me a bit
> worried.

I agree, it is worrying! But there's not much that can be done about
it: if you want to make a single wheel that works across lots of linux
distributions, and you need openssl, then you need to ship openssl,
and that means that you need to take responsibility for keeping an eye
out for security releases and respin your wheels when they happen.

I guess the psycopg2 developers are already doing this for their
Windows wheels, so perhaps they'll be willing to take on that
responsibility for Linux wheels too? Or there might be things we can
do to mitigate this somewhat: e.g., one can imagine the different
folks who need to ship openssl getting together and building some
shared infrastructure, so that they can share the burden -- like have
one set of build scripts to update openssl and rebuild all their
packages. Or once the native-libraries-in-wheels effort gets off the
ground (sorry, I've been meaning to send out an email to kick that off
but have been distracted by other things), then they'll be able to
share a single openssl wheel and then that'll also centralize the
update process. But fundamentally, the unavoidable fact is that if
someone's not willing to take on responsibility for this somehow, then
probably it's a bad idea for that person to ship manylinux wheels (or
windows wheels, for that matter).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org