[Wheel-builders] libpythonX.Y.so.1

[Nate Coraor]

On Wed, Mar 2, 2016 at 11:05 AM, Nate Coraor <nate at bx.psu.edu> wrote:

> On Tue, Mar 1, 2016 at 4:58 PM, Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Tue, Mar 1, 2016 at 11:18 AM, Nate Coraor <nate at bx.psu.edu> wrote:
>> > So I am working on building my first set of manylinux1 wheels. The first
>> > candidate is psycopg2. I was able to generate the wheel which brings in
>> all
>> > the necessary libs:
>> [...]
>> > However, attempting to use this wheel on Ubuntu 14.04 results in a
>> failure
>> > to load libpython2.7.so.1:
>> >
>> >   ImportError: libpython2.7.so.1.0: cannot open shared object file: No
>> such
>> > file or directory
>> >
>> > PEP 513 addresses libpythonX.Y.so.1 but only to say that it does not
>> need to
>> > be linked, nor should it be. Is my understanding correct, then, that I
>> would
>> > need to fix psycopg2 myself to *not* link to libpython (if this works -
>> > won't there be unresolved symbols at link time?)? Is there a more
>> general
>> > solution to this problem?
>>
>> Right, you need to not link to libpython. This does work -- it turns
>> out that the linker doesn't check for unresolved symbols unless you
>> specifically ask it to (via -Wl,--no-undefined or similar), I guess
>> because the people who write linkers consider what we're doing here to
>> be a valid use case :-).
>>
>> Are you using the docker images to build these? I thought we fixed it
>> to not link to libpython by default, but if not then we should... (If
>> your python is built with --disable-shared, which is the default when
>> building from source, then by default any wheels you build with that
>> python will not link to libpython. Unfortunately CentOS's python is
>> built with --enable-shared. This is one of the reasons that the docker
>> images build their own python instead of using the one shipped by
>> CentOS.)
>>
>
> I'm using the docker image and the purpose-built Python, yeah. It's a
> container that I started a few weeks ago though, so I can pull a new
> version of the image and see if it's fixed. The Python in my version of the
> image definitely has a shared libpython in the purpose-built Python build:
>
> libpython2.7.so.1.0 => /opt/2.7.11/lib/libpython2.7.so.1.0
> (0x00007fab9bdfd000)
>

Yeah, sure enough, this is just due to an out-of-date image. The one I was
using only had the shared libpython, the latest one only has the static. A
psycopg2 wheel built in the new image is working.

Two further questions now that this works for me:

1. With SOABI tag support now available for Python 2.X, would a PR adding
UCS-2 builds of Python to the docker image be accepted?

2. Is anyone working on the pip side of the changes necessary to install
manylinux1 wheels? If not, I'll do this.

Also, I wonder if we should vendor auditwheel into wheel itself so you can
directly build manylinux1 wheels.


>
>>
>> Also, I guess Robert is a bit distracted by thesis-ing, but I think he
>> was at least planning to make auditwheel check for and reject wheels
>> that link to libpython (or possibly repair them -- I think patchelf
>> can do that). Perhaps he'll speak up about that, or I'm sure he'd
>> welcome a pull request :-). I guess that's the more-general solution
>> you're looking for.
>>
>
> A good idea, but if the docker image isn't supposed to produce wheels that
> link libpython, that's also a good general solution for me. =)
>
>
>>
>> > Related, some of these are concerning to me. Although CentOS 5 receives
>> > updates, so its OpenSSL should be secure and bug-free, this is still
>> OpenSSL
>> > 0.9.8e. And bundling SELinux and Kerberos/GSSAPI libs also makes me a
>> bit
>> > worried.
>>
>> I agree, it is worrying! But there's not much that can be done about
>> it: if you want to make a single wheel that works across lots of linux
>> distributions, and you need openssl, then you need to ship openssl,
>> and that means that you need to take responsibility for keeping an eye
>> out for security releases and respin your wheels when they happen.
>>
>> I guess the psycopg2 developers are already doing this for their
>> Windows wheels, so perhaps they'll be willing to take on that
>> responsibility for Linux wheels too? Or there might be things we can
>> do to mitigate this somewhat: e.g., one can imagine the different
>> folks who need to ship openssl getting together and building some
>> shared infrastructure, so that they can share the burden -- like have
>> one set of build scripts to update openssl and rebuild all their
>> packages. Or once the native-libraries-in-wheels effort gets off the
>> ground (sorry, I've been meaning to send out an email to kick that off
>> but have been distracted by other things), then they'll be able to
>> share a single openssl wheel and then that'll also centralize the
>> update process. But fundamentally, the unavoidable fact is that if
>> someone's not willing to take on responsibility for this somehow, then
>> probably it's a bad idea for that person to ship manylinux wheels (or
>> windows wheels, for that matter).
>>
>
> Sure, this makes sense to me. I wonder if it would be a good idea to have
> something like Conda in the docker image that would make installing
> up-to-date versions of these sorts of dependencies trivial.
>
> Thanks,
> --nate
>
> P.S., the archives for this list aren't showing up on mail.python.org,
> could you take a look in the list admin settings and see if they're enabled?
>
>
>
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20160302/7f58f125/attachment.html>