From eric.noulard at gmail.com  Tue Feb 18 03:29:13 2020
From: eric.noulard at gmail.com (Eric Noulard)
Date: Tue, 18 Feb 2020 09:29:13 +0100
Subject: [Wheel-builders] Trouble using manylinux1 docker image in a Gitlab
 CI runner
Message-ID: <CAPASGLp2KBTdynuJcAqUEzjx+n7RPfSbDrKmJVUw8vEwww+XPA@mail.gmail.com>

Hi there,

Thank you very much for providing docker images for building manylinux
wheels.
We use those images in our in-house Gitlab CI instance and I had trouble
with:

quay.io/pypa/manylinux1_x86_64
whereas
quay.io/pypa/manylinux2010_x86_64 and quay.io/pypa/manylinux2014_x86_64

This looks like a Gitlab runner bug, so I filed one:
https://gitlab.com/gitlab-org/gitlab-runner/issues/8308

Does anyone here has similar issue?

-- 
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200218/88348db2/attachment.html>

From dalley at redhat.com  Mon Feb 17 13:50:34 2020
From: dalley at redhat.com (Daniel Alley)
Date: Mon, 17 Feb 2020 13:50:34 -0500
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
Message-ID: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>

Hi all,

I am attempting to package a C library with python extensions,
createrepo_c, as a Python wheel that we can distribute on PyPI.  The
library compiles successfully on every platform we've tried, but
"auditwheel repair" fails with the error:

auditwheel: error: cannot repair
> "wheelhouse/createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl" to
> "manylinux2014_x86_64" ABI because of the presence of too-recent versioned
> symbols. You'll need to compile the wheel on an older toolchain.
>

Here is the output of "auditwheel show" and "auditwheel repair":
https://pastebin.com/zBA2KGqV

I *think* (but don't know for sure) that the problem is that installing the
development headers for libcurl forces new versions of curl and libcurl to
be installed.  But I'm not that experienced at this and the tooling doesn't
explicitly say which symbols are "too recent".

https://pastebin.com/raw/1uDxrwXx

Here is the full code -- see the branch referred to by this PR:
https://github.com/rpm-software-management/createrepo_c/pull/207

Can anybody point me in the right direction towards resolving this issue?
Is this a bug, a problem with my own code, a problem with the RPMs
available?

Thank you,
Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200217/a05dfab8/attachment.html>

From njs at pobox.com  Tue Feb 18 08:33:32 2020
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 18 Feb 2020 05:33:32 -0800
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
Message-ID: <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>

On Tue, Feb 18, 2020 at 5:19 AM Daniel Alley <dalley at redhat.com> wrote:
>
> Hi all,
>
> I am attempting to package a C library with python extensions, createrepo_c, as a Python wheel that we can distribute on PyPI.  The library compiles successfully on every platform we've tried, but "auditwheel repair" fails with the error:
>
>> auditwheel: error: cannot repair "wheelhouse/createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl" to "manylinux2014_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.
>
>
> Here is the output of "auditwheel show" and "auditwheel repair":  https://pastebin.com/zBA2KGqV
>
> I *think* (but don't know for sure) that the problem is that installing the development headers for libcurl forces new versions of curl and libcurl to be installed.  But I'm not that experienced at this and the tooling doesn't explicitly say which symbols are "too recent".
>
> https://pastebin.com/raw/1uDxrwXx
>
> Here is the full code -- see the branch referred to by this PR: https://github.com/rpm-software-management/createrepo_c/pull/207
>
> Can anybody point me in the right direction towards resolving this issue?  Is this a bug, a problem with my own code, a problem with the RPMs available?

I don't think libcurl is involved ? 'auditwheel repair' will vendor
libcurl. The libraries it can't vendor are the C runtime libraries
like libc, libm, libgcc_s. (I agree that it'd be nice if auditwheel
gave more details!)

Are you using the manylinux2014 docker images to build your wheels?
That should automatically take care of the C runtime issues and
various other subtle details. It's not technically mandatory, but in
practice basically everyone should be using it.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From dalley at redhat.com  Tue Feb 18 09:07:54 2020
From: dalley at redhat.com (Daniel Alley)
Date: Tue, 18 Feb 2020 09:07:54 -0500
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
 <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
Message-ID: <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>

>
> Are you using the manylinux2014 docker images to build your wheels?
> That should automatically take care of the C runtime issues and
> various other subtle details. It's not technically mandatory, but in
> practice basically everyone should be using it.
>

I'm using the official docker images from quay.io and the script provided
here: https://github.com/pypa/python-manylinux-demo/tree/master/travis

Modified as such:

#!/bin/bash
> set -e -x
>
> # Install a system package required by our library
> yum install -y cmake make bzip2-devel expat-devel file-devel glib2-devel
> libcurl-devel libmodulemd2-devel libxml2-devel openssl-devel sqlite-devel
> zchunk-devel zlib-devel xz-devel rpm-devel
>
> # Compile wheels
> for PYBIN in /opt/python/*/bin; do
>     "${PYBIN}/pip" install -r /io/requirements-dev.txt
>     "${PYBIN}/pip" wheel /io/ -w wheelhouse/
> done
>
> # Bundle external shared libraries into the wheels
> for whl in wheelhouse/*.whl; do
>     auditwheel show "$whl"
>     auditwheel repair "$whl" --plat $PLAT -w /io/wheelhouse/
> done
>
> # Install packages and test
> for PYBIN in /opt/python/*/bin/; do
>     "${PYBIN}/pip" install createrepo_c --no-index -f /io/wheelhouse
>     (cd "$HOME"; "${PYBIN}/nosetests" createrepo_c)
> done
>

I suppose it's worth mentioning that libmodulemd2 is a gobject-based
library, do you think that might present problems?

On Tue, Feb 18, 2020 at 8:37 AM Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Feb 18, 2020 at 5:19 AM Daniel Alley <dalley at redhat.com> wrote:
> >
> > Hi all,
> >
> > I am attempting to package a C library with python extensions,
> createrepo_c, as a Python wheel that we can distribute on PyPI.  The
> library compiles successfully on every platform we've tried, but
> "auditwheel repair" fails with the error:
> >
> >> auditwheel: error: cannot repair
> "wheelhouse/createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl" to
> "manylinux2014_x86_64" ABI because of the presence of too-recent versioned
> symbols. You'll need to compile the wheel on an older toolchain.
> >
> >
> > Here is the output of "auditwheel show" and "auditwheel repair":
> https://pastebin.com/zBA2KGqV
> >
> > I *think* (but don't know for sure) that the problem is that installing
> the development headers for libcurl forces new versions of curl and libcurl
> to be installed.  But I'm not that experienced at this and the tooling
> doesn't explicitly say which symbols are "too recent".
> >
> > https://pastebin.com/raw/1uDxrwXx
> >
> > Here is the full code -- see the branch referred to by this PR:
> https://github.com/rpm-software-management/createrepo_c/pull/207
> >
> > Can anybody point me in the right direction towards resolving this
> issue?  Is this a bug, a problem with my own code, a problem with the RPMs
> available?
>
> I don't think libcurl is involved ? 'auditwheel repair' will vendor
> libcurl. The libraries it can't vendor are the C runtime libraries
> like libc, libm, libgcc_s. (I agree that it'd be nice if auditwheel
> gave more details!)
>
> Are you using the manylinux2014 docker images to build your wheels?
> That should automatically take care of the C runtime issues and
> various other subtle details. It's not technically mandatory, but in
> practice basically everyone should be using it.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200218/b5afcc2d/attachment-0001.html>

From njs at pobox.com  Tue Feb 18 09:45:22 2020
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 18 Feb 2020 06:45:22 -0800
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
 <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
 <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>
Message-ID: <CAPJVwBn1nmj0rzCNJcuLwoYYEMP0CR=Mfy8fdY1iEn_g3QiCgw@mail.gmail.com>

On Tue, Feb 18, 2020 at 6:08 AM Daniel Alley <dalley at redhat.com> wrote:
> I suppose it's worth mentioning that libmodulemd2 is a gobject-based library, do you think that might present problems?

No. Somehow you're not using the compiler or libc that come inside the
docker image. Individual libraries can't cause these kinds of issues.
So either you're somehow not using the docker image, or you're somehow
replacing the entire OS inside the docker image, or ... I can't think
of any other possibilities :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From dalley at redhat.com  Tue Feb 18 10:12:34 2020
From: dalley at redhat.com (Daniel Alley)
Date: Tue, 18 Feb 2020 10:12:34 -0500
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAPJVwBn1nmj0rzCNJcuLwoYYEMP0CR=Mfy8fdY1iEn_g3QiCgw@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
 <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
 <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>
 <CAPJVwBn1nmj0rzCNJcuLwoYYEMP0CR=Mfy8fdY1iEn_g3QiCgw@mail.gmail.com>
Message-ID: <CAK1D4mDRZM-X0JEXa=Y2HtR+d+dVnRZNXUkK499yexxZqzuKXA@mail.gmail.com>

I'm not sure how that can be happening then.  I'm using this exact setup to
build packages for 2 other libraries and both of those packages work great
-- this is the only one causing issues.  It has the most dependencies out
of the three.

Here's the command I'm using from the host, and you can see from the build
script that I'm not installing any new compiler.

docker run --rm -e PLAT=manylinux2014_x86_64 -v `pwd`:/io
> quay.io/pypa/manylinux2014_x86_64 /io/build_wheels.sh
>

On Tue, Feb 18, 2020 at 9:45 AM Nathaniel Smith <njs at pobox.com> wrote:

> On Tue, Feb 18, 2020 at 6:08 AM Daniel Alley <dalley at redhat.com> wrote:
> > I suppose it's worth mentioning that libmodulemd2 is a gobject-based
> library, do you think that might present problems?
>
> No. Somehow you're not using the compiler or libc that come inside the
> docker image. Individual libraries can't cause these kinds of issues.
> So either you're somehow not using the docker image, or you're somehow
> replacing the entire OS inside the docker image, or ... I can't think
> of any other possibilities :-).
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200218/a4edd9b2/attachment.html>

From dalley at redhat.com  Tue Feb 18 11:39:36 2020
From: dalley at redhat.com (Daniel Alley)
Date: Tue, 18 Feb 2020 11:39:36 -0500
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAK1D4mDRZM-X0JEXa=Y2HtR+d+dVnRZNXUkK499yexxZqzuKXA@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
 <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
 <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>
 <CAPJVwBn1nmj0rzCNJcuLwoYYEMP0CR=Mfy8fdY1iEn_g3QiCgw@mail.gmail.com>
 <CAK1D4mDRZM-X0JEXa=Y2HtR+d+dVnRZNXUkK499yexxZqzuKXA@mail.gmail.com>
Message-ID: <CAK1D4mBZ9A4aLLLG61BqLfee+QFUKnQiPvf63kP86CLo07-qcQ@mail.gmail.com>

So, I just went through the "auditwheel show" output and compared it
against the manylinux2014 symbol version whitelists from here:
https://github.com/pypa/auditwheel/blob/8b255b5cf09365d2fd4ab770996005df05a62b63/auditwheel/policy/policy.json

All of the versions listed for GCC and GLIBC are in the whitelist, except
for one which seems out of place.  What is GLIB_PRIVATE?

libc.so.6 with versions {'GLIBC_2.4','GLIBC_2.15', 'GLIBC_2.3.3',
> 'GLIBC_2.8', 'GLIBC_PRIVATE', 'GLIBC_2.7', 'GLIBC_2.14', 'GLIBC_2.3',
> 'GLIBC_2.2.5', 'GLIBC_2.16', 'GLIBC_2.3.2', 'GLIBC_2.17', 'GLIBC_2.3.4',
> 'GLIBC_2.11', 'GLIBC_2.12'},
>

Apart from this, I'm not really sure what to be looking for.

On Tue, Feb 18, 2020 at 10:12 AM Daniel Alley <dalley at redhat.com> wrote:

> I'm not sure how that can be happening then.  I'm using this exact setup
> to build packages for 2 other libraries and both of those packages work
> great -- this is the only one causing issues.  It has the most dependencies
> out of the three.
>
> Here's the command I'm using from the host, and you can see from the build
> script that I'm not installing any new compiler.
>
> docker run --rm -e PLAT=manylinux2014_x86_64 -v `pwd`:/io
>> quay.io/pypa/manylinux2014_x86_64 /io/build_wheels.sh
>>
>
> On Tue, Feb 18, 2020 at 9:45 AM Nathaniel Smith <njs at pobox.com> wrote:
>
>> On Tue, Feb 18, 2020 at 6:08 AM Daniel Alley <dalley at redhat.com> wrote:
>> > I suppose it's worth mentioning that libmodulemd2 is a gobject-based
>> library, do you think that might present problems?
>>
>> No. Somehow you're not using the compiler or libc that come inside the
>> docker image. Individual libraries can't cause these kinds of issues.
>> So either you're somehow not using the docker image, or you're somehow
>> replacing the entire OS inside the docker image, or ... I can't think
>> of any other possibilities :-).
>>
>> -n
>>
>> --
>> Nathaniel J. Smith -- https://vorpus.org
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200218/aa6fe852/attachment.html>

From njs at pobox.com  Tue Feb 18 11:45:09 2020
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 18 Feb 2020 08:45:09 -0800
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAK1D4mBZ9A4aLLLG61BqLfee+QFUKnQiPvf63kP86CLo07-qcQ@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
 <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
 <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>
 <CAPJVwBn1nmj0rzCNJcuLwoYYEMP0CR=Mfy8fdY1iEn_g3QiCgw@mail.gmail.com>
 <CAK1D4mDRZM-X0JEXa=Y2HtR+d+dVnRZNXUkK499yexxZqzuKXA@mail.gmail.com>
 <CAK1D4mBZ9A4aLLLG61BqLfee+QFUKnQiPvf63kP86CLo07-qcQ@mail.gmail.com>
Message-ID: <CAPJVwBk1qxh+gTtsEgGLEPr_2ZvR8dAM44nwjKLwO1aBS6q_6w@mail.gmail.com>

Huh. I'm not sure what GLIBC_PRIVATE is, but it doesn't seem like something
your wheel *should* be depending on... So either there's something funny
about your exact library setup, or there's a bug in how auditwheel is
looking up symbol versions. I think at this point I'd file a bug on
auditwheel.

On Tue, Feb 18, 2020, 08:40 Daniel Alley <dalley at redhat.com> wrote:

> So, I just went through the "auditwheel show" output and compared it
> against the manylinux2014 symbol version whitelists from here:
> https://github.com/pypa/auditwheel/blob/8b255b5cf09365d2fd4ab770996005df05a62b63/auditwheel/policy/policy.json
>
> All of the versions listed for GCC and GLIBC are in the whitelist, except
> for one which seems out of place.  What is GLIB_PRIVATE?
>
> libc.so.6 with versions {'GLIBC_2.4','GLIBC_2.15', 'GLIBC_2.3.3',
>> 'GLIBC_2.8', 'GLIBC_PRIVATE', 'GLIBC_2.7', 'GLIBC_2.14', 'GLIBC_2.3',
>> 'GLIBC_2.2.5', 'GLIBC_2.16', 'GLIBC_2.3.2', 'GLIBC_2.17', 'GLIBC_2.3.4',
>> 'GLIBC_2.11', 'GLIBC_2.12'},
>>
>
> Apart from this, I'm not really sure what to be looking for.
>
> On Tue, Feb 18, 2020 at 10:12 AM Daniel Alley <dalley at redhat.com> wrote:
>
>> I'm not sure how that can be happening then.  I'm using this exact setup
>> to build packages for 2 other libraries and both of those packages work
>> great -- this is the only one causing issues.  It has the most dependencies
>> out of the three.
>>
>> Here's the command I'm using from the host, and you can see from the
>> build script that I'm not installing any new compiler.
>>
>> docker run --rm -e PLAT=manylinux2014_x86_64 -v `pwd`:/io
>>> quay.io/pypa/manylinux2014_x86_64 /io/build_wheels.sh
>>>
>>
>> On Tue, Feb 18, 2020 at 9:45 AM Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> On Tue, Feb 18, 2020 at 6:08 AM Daniel Alley <dalley at redhat.com> wrote:
>>> > I suppose it's worth mentioning that libmodulemd2 is a gobject-based
>>> library, do you think that might present problems?
>>>
>>> No. Somehow you're not using the compiler or libc that come inside the
>>> docker image. Individual libraries can't cause these kinds of issues.
>>> So either you're somehow not using the docker image, or you're somehow
>>> replacing the entire OS inside the docker image, or ... I can't think
>>> of any other possibilities :-).
>>>
>>> -n
>>>
>>> --
>>> Nathaniel J. Smith -- https://vorpus.org
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200218/252876b0/attachment-0001.html>

From dalley at redhat.com  Tue Feb 18 12:06:23 2020
From: dalley at redhat.com (Daniel Alley)
Date: Tue, 18 Feb 2020 12:06:23 -0500
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAPJVwBk1qxh+gTtsEgGLEPr_2ZvR8dAM44nwjKLwO1aBS6q_6w@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
 <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
 <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>
 <CAPJVwBn1nmj0rzCNJcuLwoYYEMP0CR=Mfy8fdY1iEn_g3QiCgw@mail.gmail.com>
 <CAK1D4mDRZM-X0JEXa=Y2HtR+d+dVnRZNXUkK499yexxZqzuKXA@mail.gmail.com>
 <CAK1D4mBZ9A4aLLLG61BqLfee+QFUKnQiPvf63kP86CLo07-qcQ@mail.gmail.com>
 <CAPJVwBk1qxh+gTtsEgGLEPr_2ZvR8dAM44nwjKLwO1aBS6q_6w@mail.gmail.com>
Message-ID: <CAK1D4mDgyFrmj+1TepCq0hmgZ-3g=81mP6b=x3rsgGjTMXCttQ@mail.gmail.com>

Filed here: https://github.com/pypa/auditwheel/issues/229

On Tue, Feb 18, 2020 at 11:45 AM Nathaniel Smith <njs at pobox.com> wrote:

> Huh. I'm not sure what GLIBC_PRIVATE is, but it doesn't seem like
> something your wheel *should* be depending on... So either there's
> something funny about your exact library setup, or there's a bug in how
> auditwheel is looking up symbol versions. I think at this point I'd file a
> bug on auditwheel.
>
> On Tue, Feb 18, 2020, 08:40 Daniel Alley <dalley at redhat.com> wrote:
>
>> So, I just went through the "auditwheel show" output and compared it
>> against the manylinux2014 symbol version whitelists from here:
>> https://github.com/pypa/auditwheel/blob/8b255b5cf09365d2fd4ab770996005df05a62b63/auditwheel/policy/policy.json
>>
>> All of the versions listed for GCC and GLIBC are in the whitelist, except
>> for one which seems out of place.  What is GLIB_PRIVATE?
>>
>> libc.so.6 with versions {'GLIBC_2.4','GLIBC_2.15', 'GLIBC_2.3.3',
>>> 'GLIBC_2.8', 'GLIBC_PRIVATE', 'GLIBC_2.7', 'GLIBC_2.14', 'GLIBC_2.3',
>>> 'GLIBC_2.2.5', 'GLIBC_2.16', 'GLIBC_2.3.2', 'GLIBC_2.17', 'GLIBC_2.3.4',
>>> 'GLIBC_2.11', 'GLIBC_2.12'},
>>>
>>
>> Apart from this, I'm not really sure what to be looking for.
>>
>> On Tue, Feb 18, 2020 at 10:12 AM Daniel Alley <dalley at redhat.com> wrote:
>>
>>> I'm not sure how that can be happening then.  I'm using this exact setup
>>> to build packages for 2 other libraries and both of those packages work
>>> great -- this is the only one causing issues.  It has the most dependencies
>>> out of the three.
>>>
>>> Here's the command I'm using from the host, and you can see from the
>>> build script that I'm not installing any new compiler.
>>>
>>> docker run --rm -e PLAT=manylinux2014_x86_64 -v `pwd`:/io
>>>> quay.io/pypa/manylinux2014_x86_64 /io/build_wheels.sh
>>>>
>>>
>>> On Tue, Feb 18, 2020 at 9:45 AM Nathaniel Smith <njs at pobox.com> wrote:
>>>
>>>> On Tue, Feb 18, 2020 at 6:08 AM Daniel Alley <dalley at redhat.com> wrote:
>>>> > I suppose it's worth mentioning that libmodulemd2 is a gobject-based
>>>> library, do you think that might present problems?
>>>>
>>>> No. Somehow you're not using the compiler or libc that come inside the
>>>> docker image. Individual libraries can't cause these kinds of issues.
>>>> So either you're somehow not using the docker image, or you're somehow
>>>> replacing the entire OS inside the docker image, or ... I can't think
>>>> of any other possibilities :-).
>>>>
>>>> -n
>>>>
>>>> --
>>>> Nathaniel J. Smith -- https://vorpus.org
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200218/92b17e59/attachment.html>

From dalley at redhat.com  Tue Feb 18 13:31:17 2020
From: dalley at redhat.com (Daniel Alley)
Date: Tue, 18 Feb 2020 13:31:17 -0500
Subject: [Wheel-builders] Trouble building manylinux2014 wheel
In-Reply-To: <CAK1D4mDgyFrmj+1TepCq0hmgZ-3g=81mP6b=x3rsgGjTMXCttQ@mail.gmail.com>
References: <CAK1D4mB8xWKW=-FjW_RDrf2wjcum1d_8tavz_ij0MjEFZmNyUg@mail.gmail.com>
 <CAPJVwBnGKa3YPJp7rFyH5EHmoHxSbMfHRbwnhKCXs-EhnqigYA@mail.gmail.com>
 <CAK1D4mBfSKBm5g=HqjmLxN5fwcXUeBcEH8RCpEA9CtyujB7MrQ@mail.gmail.com>
 <CAPJVwBn1nmj0rzCNJcuLwoYYEMP0CR=Mfy8fdY1iEn_g3QiCgw@mail.gmail.com>
 <CAK1D4mDRZM-X0JEXa=Y2HtR+d+dVnRZNXUkK499yexxZqzuKXA@mail.gmail.com>
 <CAK1D4mBZ9A4aLLLG61BqLfee+QFUKnQiPvf63kP86CLo07-qcQ@mail.gmail.com>
 <CAPJVwBk1qxh+gTtsEgGLEPr_2ZvR8dAM44nwjKLwO1aBS6q_6w@mail.gmail.com>
 <CAK1D4mDgyFrmj+1TepCq0hmgZ-3g=81mP6b=x3rsgGjTMXCttQ@mail.gmail.com>
Message-ID: <CAK1D4mChx82syB8932EGTBJimou2RwE7B9-NgMK9qKuGM0Uw-A@mail.gmail.com>

On a semi-unrelated note, I'm also interested in packaging libmodulemd
itself, which is (as I mentioned) a gobject-based library.  Is it possible
to make that packaged in a self-contained way, without having a ton of
system dependencies?

On Tue, Feb 18, 2020 at 12:06 PM Daniel Alley <dalley at redhat.com> wrote:

> Filed here: https://github.com/pypa/auditwheel/issues/229
>
> On Tue, Feb 18, 2020 at 11:45 AM Nathaniel Smith <njs at pobox.com> wrote:
>
>> Huh. I'm not sure what GLIBC_PRIVATE is, but it doesn't seem like
>> something your wheel *should* be depending on... So either there's
>> something funny about your exact library setup, or there's a bug in how
>> auditwheel is looking up symbol versions. I think at this point I'd file a
>> bug on auditwheel.
>>
>> On Tue, Feb 18, 2020, 08:40 Daniel Alley <dalley at redhat.com> wrote:
>>
>>> So, I just went through the "auditwheel show" output and compared it
>>> against the manylinux2014 symbol version whitelists from here:
>>> https://github.com/pypa/auditwheel/blob/8b255b5cf09365d2fd4ab770996005df05a62b63/auditwheel/policy/policy.json
>>>
>>> All of the versions listed for GCC and GLIBC are in the whitelist,
>>> except for one which seems out of place.  What is GLIB_PRIVATE?
>>>
>>> libc.so.6 with versions {'GLIBC_2.4','GLIBC_2.15', 'GLIBC_2.3.3',
>>>> 'GLIBC_2.8', 'GLIBC_PRIVATE', 'GLIBC_2.7', 'GLIBC_2.14', 'GLIBC_2.3',
>>>> 'GLIBC_2.2.5', 'GLIBC_2.16', 'GLIBC_2.3.2', 'GLIBC_2.17', 'GLIBC_2.3.4',
>>>> 'GLIBC_2.11', 'GLIBC_2.12'},
>>>>
>>>
>>> Apart from this, I'm not really sure what to be looking for.
>>>
>>> On Tue, Feb 18, 2020 at 10:12 AM Daniel Alley <dalley at redhat.com> wrote:
>>>
>>>> I'm not sure how that can be happening then.  I'm using this exact
>>>> setup to build packages for 2 other libraries and both of those packages
>>>> work great -- this is the only one causing issues.  It has the most
>>>> dependencies out of the three.
>>>>
>>>> Here's the command I'm using from the host, and you can see from the
>>>> build script that I'm not installing any new compiler.
>>>>
>>>> docker run --rm -e PLAT=manylinux2014_x86_64 -v `pwd`:/io
>>>>> quay.io/pypa/manylinux2014_x86_64 /io/build_wheels.sh
>>>>>
>>>>
>>>> On Tue, Feb 18, 2020 at 9:45 AM Nathaniel Smith <njs at pobox.com> wrote:
>>>>
>>>>> On Tue, Feb 18, 2020 at 6:08 AM Daniel Alley <dalley at redhat.com>
>>>>> wrote:
>>>>> > I suppose it's worth mentioning that libmodulemd2 is a gobject-based
>>>>> library, do you think that might present problems?
>>>>>
>>>>> No. Somehow you're not using the compiler or libc that come inside the
>>>>> docker image. Individual libraries can't cause these kinds of issues.
>>>>> So either you're somehow not using the docker image, or you're somehow
>>>>> replacing the entire OS inside the docker image, or ... I can't think
>>>>> of any other possibilities :-).
>>>>>
>>>>> -n
>>>>>
>>>>> --
>>>>> Nathaniel J. Smith -- https://vorpus.org
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200218/feaf9365/attachment.html>

From dalley at redhat.com  Sun Feb 23 19:30:33 2020
From: dalley at redhat.com (Daniel Alley)
Date: Sun, 23 Feb 2020 19:30:33 -0500
Subject: [Wheel-builders] Packaging a library that uses PyGObject?
Message-ID: <CAK1D4mCEmRg2TxusDUdO3YSYBCbN9L5O2ZuwE6fxG94krA0+5A@mail.gmail.com>

I would like to package this library as a pre-built Python wheel:
https://github.com/fedora-modularity/libmodulemd

This library uses PyGObject, so importing it looks like this:

> import gi
> gi.require_version("Modulemd", "2.0")from gi.repository import Modulemd
>
>  I can't find any examples of this, nor any documentation, nor discussion
about it.  It seems like it would be, at minimum, a bit more complicated
than libraries based on normal bindings.  Is this possible, and are there
any special requirements that are needed to do so?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200223/eaefd206/attachment.html>

From geofft at ldpreload.com  Sun Feb 23 20:26:54 2020
From: geofft at ldpreload.com (Geoffrey Thomas)
Date: Sun, 23 Feb 2020 20:26:54 -0500 (EST)
Subject: [Wheel-builders] Packaging a library that uses PyGObject?
In-Reply-To: <CAK1D4mCEmRg2TxusDUdO3YSYBCbN9L5O2ZuwE6fxG94krA0+5A@mail.gmail.com>
References: <CAK1D4mCEmRg2TxusDUdO3YSYBCbN9L5O2ZuwE6fxG94krA0+5A@mail.gmail.com>
Message-ID: <alpine.DEB.2.11.2002231950380.4589@titan.ldpreload.com>

On Sun, 23 Feb 2020, Daniel Alley wrote:

> I would like to package this library as a pre-built Python wheel: https://github.com/fedora-modularity/libmodulemd
> 
> This library uses PyGObject, so importing it looks like this:
> 
> import gi
> gi.require_version("Modulemd", "2.0")
> from gi.repository import Modulemd
> 
> ?I can't find any examples of this, nor any documentation, nor discussion about it.? It seems like it would be, at minimum, a bit more complicated than libraries based on normal bindings.? Is this
> possible, and are there any special requirements that are needed to do so??

It's been a bit since I've used gobject-introspection, but I _think_ the 
way this works for a normal (OS-installed) GObject package is

a) the package ordinarily provides no actual Python library
b) the package provides a girepository-1.0/Modulemd-2.0.typelib file in 
the system lib directory
c) when you import it, PyGObject loads the C libmodulemd library and 
generates Python bindings based on the typelib file
d) as a special case, a package _can_ provide a Python "override" library, 
but that amends the autogenerated bindings, it's not a complete set of 
bindings on its own (and it's Python code, not native code). Modulemd 
appears to do this.

So your users' code needs to be able to
- import PyGObject (and libgobject) itself, either from the system or from 
their virtualenv
- import the C libmodulemd library from your wheel, which you can compile 
for manylinux1
- find the typelib file, which you can put in your wheel
- point libgobject at the typelib file and the C library
- point PyGObject at the override file (which should hopefully be 
automatic if it's on sys.path)

The GObject docs https://developer.gnome.org/gi/stable/GIRepository.html 
say:

> GIRepository will typically look for a girepository-1.0 directory under 
> the library directory used when compiling gobject-introspection.
>
> It is possible to control the search paths programmatically, using 
> g_irepository_prepend_search_path(). It is also possible to modify the 
> search paths by using the GI_TYPELIB_PATH environment variable. The 
> environment variable takes precedence over the default search path and 
> the g_irepository_prepend_search_path() calls.

You will also need to make sure the C library is importable. For "normal" 
Python wheels, users import a compiled Python shared object (so that the 
usual Python path is used as the search path), and that shared object has 
a normal shared object dependency on the underlying C library which is 
also shipped in the wheel.auditwheel sets an $ORIGIN-relative rpath in 
that .so file (using patchelf) so that the Python module, having been 
found inside the virtualenv, can find its C library in a relative path to 
its own location. Since there is no compiled Python module in your case, 
because PyGObject is dynamically generating the bindings at runtime, I 
don't think there is a straightforward way of informing PyGObject of where 
to find the C library.

Personally, I'd approach this by first aiming for an 80% solution where I 
expect users to set GI_TYPELIB_PATH and LD_LIBRARY_PATH so that the 
typelib file and the C library can both be found, i.e., they use it by 
running something to the effect of

os.setenv("GI_TYPELIB_PATH", "myvenv/lib/girepository-1.0")
os.setenv("LD_LIBRARY_PATH", "myvenv/lib")
gi.require_version("Modulemd", "2.0")
from gi.repository import Modulemd

That would let me confirm that I've actually gotten all the libraries 
compiling properly inside the wheel and the code actually works. Then 
there's a question of how to do this automatically. A 90% solution would 
be to just decide that your wheel has a top-level Python module to do 
this, e.g., you tell your users that if they're using the wheel they just 
do "import Modulemd" and you create a Modulemd.py that does

os.setenv("GI_TYPELIB_PATH", some relative path from __file__)
etc.

(For bonus points, call g_irepository_prepend_search_path() / see if 
PyGObject has some binding to it, instead of setting $GI_TYPELIB_PATH, and 
use ctypes to load the actual C library using RTLD_GLOBAL so that it's 
already loaded when PyGObject goes looking for it, instead of setting 
$LD_LIBRARY_PATH.)

In my (naive) opinion, a 100% solution here would be teaching PyGObject 
how to find both typelib files and C libraries in paths based on sys.path, 
and then your users could use the standard upstream import instructions 
unmodified. (Actually, it's possible PyGObject does this already, but I 
don't immediately see anything about it in the docs, and my assumption is 
if you can't find examples of others doing this, the use case hasn't come 
up.)

One other question is whether your users are importing libgobject from the 
system or from a wheel. For the average desktop Linux user, it's probably 
fine to get libgobject from the system (and probably _preferable_ - you 
likely want the same version as the Gtk/GNOME/etc. libraries they might 
import, and if they're importing any of those, they almost certainly want 
the system version of Gtk etc.) It appears that PyGObject is on PyPI as 
sdists only, so if you don't want to assume your users have libgobject 
installed, you may have to first fight the battle of packaging up GObject, 
GLib, etc. into wheels.

(Relatedly, I'm guessing the reason nobody has done this yet is that most 
software that supports GObject introspection is GNOME-related in some 
fashion and therefore most people want it from their OS package manager 
and not from a wheel.)

Again, it's been a while since I've worked with GObject introspection, so 
if I got something wrong, anyone should feel free to correct me :)

-- 
Geoffrey Thomas
https://ldpreload.com
geofft at ldpreload.com

From dalley at redhat.com  Sun Feb 23 20:55:39 2020
From: dalley at redhat.com (Daniel Alley)
Date: Sun, 23 Feb 2020 20:55:39 -0500
Subject: [Wheel-builders] Packaging a library that uses PyGObject?
In-Reply-To: <alpine.DEB.2.11.2002231950380.4589@titan.ldpreload.com>
References: <CAK1D4mCEmRg2TxusDUdO3YSYBCbN9L5O2ZuwE6fxG94krA0+5A@mail.gmail.com>
 <alpine.DEB.2.11.2002231950380.4589@titan.ldpreload.com>
Message-ID: <CAK1D4mDz85rgy_=8TtEVZ-JwJ6SKaaX0rpB8QAR1OyvZCzFppQ@mail.gmail.com>

That is a fantastic and extremely thorough answer, thank you!

On Sun, Feb 23, 2020 at 8:32 PM Geoffrey Thomas <geofft at ldpreload.com>
wrote:

> On Sun, 23 Feb 2020, Daniel Alley wrote:
>
> > I would like to package this library as a pre-built Python wheel:
> https://github.com/fedora-modularity/libmodulemd
> >
> > This library uses PyGObject, so importing it looks like this:
> >
> > import gi
> > gi.require_version("Modulemd", "2.0")
> > from gi.repository import Modulemd
> >
> >  I can't find any examples of this, nor any documentation, nor
> discussion about it.  It seems like it would be, at minimum, a bit more
> complicated than libraries based on normal bindings.  Is this
> > possible, and are there any special requirements that are needed to do
> so?
>
> It's been a bit since I've used gobject-introspection, but I _think_ the
> way this works for a normal (OS-installed) GObject package is
>
> a) the package ordinarily provides no actual Python library
> b) the package provides a girepository-1.0/Modulemd-2.0.typelib file in
> the system lib directory
> c) when you import it, PyGObject loads the C libmodulemd library and
> generates Python bindings based on the typelib file
> d) as a special case, a package _can_ provide a Python "override" library,
> but that amends the autogenerated bindings, it's not a complete set of
> bindings on its own (and it's Python code, not native code). Modulemd
> appears to do this.
>
> So your users' code needs to be able to
> - import PyGObject (and libgobject) itself, either from the system or from
> their virtualenv
> - import the C libmodulemd library from your wheel, which you can compile
> for manylinux1
> - find the typelib file, which you can put in your wheel
> - point libgobject at the typelib file and the C library
> - point PyGObject at the override file (which should hopefully be
> automatic if it's on sys.path)
>
> The GObject docs https://developer.gnome.org/gi/stable/GIRepository.html
> say:
>
> > GIRepository will typically look for a girepository-1.0 directory under
> > the library directory used when compiling gobject-introspection.
> >
> > It is possible to control the search paths programmatically, using
> > g_irepository_prepend_search_path(). It is also possible to modify the
> > search paths by using the GI_TYPELIB_PATH environment variable. The
> > environment variable takes precedence over the default search path and
> > the g_irepository_prepend_search_path() calls.
>
> You will also need to make sure the C library is importable. For "normal"
> Python wheels, users import a compiled Python shared object (so that the
> usual Python path is used as the search path), and that shared object has
> a normal shared object dependency on the underlying C library which is
> also shipped in the wheel.auditwheel sets an $ORIGIN-relative rpath in
> that .so file (using patchelf) so that the Python module, having been
> found inside the virtualenv, can find its C library in a relative path to
> its own location. Since there is no compiled Python module in your case,
> because PyGObject is dynamically generating the bindings at runtime, I
> don't think there is a straightforward way of informing PyGObject of where
> to find the C library.
>
> Personally, I'd approach this by first aiming for an 80% solution where I
> expect users to set GI_TYPELIB_PATH and LD_LIBRARY_PATH so that the
> typelib file and the C library can both be found, i.e., they use it by
> running something to the effect of
>
> os.setenv("GI_TYPELIB_PATH", "myvenv/lib/girepository-1.0")
> os.setenv("LD_LIBRARY_PATH", "myvenv/lib")
> gi.require_version("Modulemd", "2.0")
> from gi.repository import Modulemd
>
> That would let me confirm that I've actually gotten all the libraries
> compiling properly inside the wheel and the code actually works. Then
> there's a question of how to do this automatically. A 90% solution would
> be to just decide that your wheel has a top-level Python module to do
> this, e.g., you tell your users that if they're using the wheel they just
> do "import Modulemd" and you create a Modulemd.py that does
>
> os.setenv("GI_TYPELIB_PATH", some relative path from __file__)
> etc.
>
> (For bonus points, call g_irepository_prepend_search_path() / see if
> PyGObject has some binding to it, instead of setting $GI_TYPELIB_PATH, and
> use ctypes to load the actual C library using RTLD_GLOBAL so that it's
> already loaded when PyGObject goes looking for it, instead of setting
> $LD_LIBRARY_PATH.)
>
> In my (naive) opinion, a 100% solution here would be teaching PyGObject
> how to find both typelib files and C libraries in paths based on sys.path,
> and then your users could use the standard upstream import instructions
> unmodified. (Actually, it's possible PyGObject does this already, but I
> don't immediately see anything about it in the docs, and my assumption is
> if you can't find examples of others doing this, the use case hasn't come
> up.)
>
> One other question is whether your users are importing libgobject from the
> system or from a wheel. For the average desktop Linux user, it's probably
> fine to get libgobject from the system (and probably _preferable_ - you
> likely want the same version as the Gtk/GNOME/etc. libraries they might
> import, and if they're importing any of those, they almost certainly want
> the system version of Gtk etc.) It appears that PyGObject is on PyPI as
> sdists only, so if you don't want to assume your users have libgobject
> installed, you may have to first fight the battle of packaging up GObject,
> GLib, etc. into wheels.
>
> (Relatedly, I'm guessing the reason nobody has done this yet is that most
> software that supports GObject introspection is GNOME-related in some
> fashion and therefore most people want it from their OS package manager
> and not from a wheel.)
>
> Again, it's been a while since I've worked with GObject introspection, so
> if I got something wrong, anyone should feel free to correct me :)
>
> --
> Geoffrey Thomas
> https://ldpreload.com
> geofft at ldpreload.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20200223/c9406561/attachment-0001.html>

From njs at pobox.com  Sun Feb 23 20:57:27 2020
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 23 Feb 2020 17:57:27 -0800
Subject: [Wheel-builders] Packaging a library that uses PyGObject?
In-Reply-To: <alpine.DEB.2.11.2002231950380.4589@titan.ldpreload.com>
References: <CAK1D4mCEmRg2TxusDUdO3YSYBCbN9L5O2ZuwE6fxG94krA0+5A@mail.gmail.com>
 <alpine.DEB.2.11.2002231950380.4589@titan.ldpreload.com>
Message-ID: <CAPJVwB=3cuSnEgaq7uUoM4zw_VEJW07t_GKfdLWbJivH_RXScA@mail.gmail.com>

On Sun, Feb 23, 2020 at 5:27 PM Geoffrey Thomas <geofft at ldpreload.com> wrote:
> (For bonus points, call g_irepository_prepend_search_path() / see if
> PyGObject has some binding to it, instead of setting $GI_TYPELIB_PATH, and
> use ctypes to load the actual C library using RTLD_GLOBAL so that it's
> already loaded when PyGObject goes looking for it, instead of setting
> $LD_LIBRARY_PATH.)

IIRC, $LD_LIBRARY_PATH is only consulted once at process startup, and
changing it after that doesn't affect the current process at all. So I
think the ctypes trick would actually be mandatory. But, it doesn't
matter whether you use RTLD_GLOBAL or RTLD_LOCAL, because the loader's
soname->dso cache is always process-global, even if the dso's are
supposed to be local. (Nothing about ELF scoping is intuitive.) So you
should probably use RTLD_LOCAL to reduce the chance of unwanted
side-effects.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org