[Wheel-builders] Error from numpy wheel - any thoughts

[Nathaniel Smith]

On Tue, Mar 29, 2016 at 2:51 PM, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Mar 29, 2016 at 10:50 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> On Tue, Mar 29, 2016 at 4:29 AM, Olivier Grisel
>> <olivier.grisel at ensta.org> wrote:
>>>> You can replicate fairly simply with:
>>>>
>>>> pip install -f https://d9a97980b71d47cde94d-aae005c4999d7244ac63632f8b80e089.ssl.cf2.rackcdn.com
>>>> numpy
>>>> cd numpy/f2py/tests/src/kind
>>>> f2py -c foo.f90 -m foo
>>>> python -c 'import numpy; import foo'
>>>
>>> I can replicate. I can also force the use of the system fortran with
>>> LD_PRELOAD (on ubuntu 14.04):
>>>
>>> LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libgfortran.so.3' python -c
>>> 'import numpy; import foo'
>>
>> This one is obviously a show-stopper - I believe that it means that
>> any module compiled against gfortran (and presumably any shipped lib
>> that numpy loads) will break after numpy has been imported.
>>
>> I don't understand the linux link rules well enough to know how to
>> fix.  I guess we could rename the libraries that we ship / vendor into
>> libs? (numpy_libgfortran.so etc).
>
> Yeah, I'm not at all sure what's going on here and whether it's
> intentional or a bug in glibc, and we should figure out what's
> actually happening and why for our own understanding, but I'm pretty
> confident that this is going to be either the solution or the
> workaround :-).
>
> I guess the general approach is that auditwheel should assign a unique
> name to every vendored library. Some possible naming strategies:
>   <wheel package name>-libgfortran.so
>   <truncated sha-256 of libgfortran.so>-libgfortran.so
>   <random uuid>-libgfortran.so
>
> I guess the truncated-sha256 strategy might be best because if there
> are lots of manylinux wheels all vendoring the same copy of
> libgfortran, they will (or at least might) end up sharing it in memory
> instead of mmap'ing multiple different copies of the identical .so
> file, which is good? And this is certainly safe if the different
> copies of libgfortran all have the same sha256.
>
> And then auditwheel should use patchelf --replace-needed to modify the
> DT_NEEDED in the various extension modules to point to the new name,
> at the same time it's changing the RPATH. (--replace-needed doesn't
> seem to be mentioned in most of the patchelf docs I can find, but it
> is described here: https://github.com/NixOS/patchelf)

So generally speaking this workaround is definitely the right
workaround, and I just implemented and tested it in this repository
and it works:
  https://github.com/njsmith/rpath-weirdness

But, in the process I discovered that the text described above is
missing a crucial step -- we need to also patch the DT_SONAME of the
renamed+vendored library, as described here:
  https://github.com/pypa/auditwheel/issues/24

-n

-- 
Nathaniel J. Smith -- https://vorpus.org