[Wheel-builders] error: signalfd.h not found when building manylinux wheel

[Vitaly Kruglikov]

On 7/9/16, 11:15 PM, "Nathaniel Smith" <njs at pobox.com> wrote:

>On Fri, Jul 8, 2016 at 10:33 PM, Vitaly Kruglikov
><vkruglikov at numenta.com> wrote:
>>
>>
>> On 7/8/16, 10:07 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
>>
>>>On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov
>>><vkruglikov at numenta.com>
>>>wrote:
>>>> I am attempting to build a manylinux wheel from the nupic.core project
>>>>in
>>>> github. I am using the docker image quay.io/pypa/manylinux1_x86_64.
>>>> nupic.core builds and statically links against the capnproto library,
>>>>which
>>>> relies on signalfd.h. Unfortunately, the docker image
>>>> quay.io/pypa/manylinux1_x86_64 does not provide  signalfd.h, so my
>>>>build
>>>> fails like this:
>>>>
>>>> Linking CXX static library libkj.a
>>>> [ 27%] Built target kj
>>>> [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o
>>>> [ 30%] Building CXX object
>>>>src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o
>>>>
>>>>/nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix
>>>>.c
>>>>++:36:26:
>>>> fatal error: sys/signalfd.h: No such file or directory
>>>>  #include <sys/signalfd.h>
>>>>
>>>> What is the recommended solution for this problem?
>>>
>>>man signalfd says:
>>>
>>>VERSIONS
>>>       signalfd() is available on Linux since kernel 2.6.22.  Working
>>>support
>>>       is  provided  in  glibc since version 2.8.  The signalfd4()
>>>system
>>>call
>>>       (see NOTES) is available on Linux since kernel 2.6.27.
>>>
>>>CentOS 5 ships glibc 2.5, so signalfd is simply not available to
>>>manylinux1 wheels. I guess the recommended solution is to modify your
>>>code so that it doesn't require signalfd? It looks you aren't the only
>>>person to run into this this week...
>>>
>>>    https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU
>>>
>>>and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto
>>>not to use signalfd.
>>>
>>>This is an interesting data point for the benefits of defining a
>>>CentOS-6-based manylinux2, though...
>>>
>>>-n
>>>
>>>--
>>>Nathaniel J. Smith -- https://vorpus.org
>>
>>
>> Thanks Nathaniel, unfortunately it is not as simple as that.
>> Unfortunately, capnproto is not my code, so I am somewhat limited in
>>what
>> I can do with it. You also replied to a similar question concerning
>>pipe2,
>> SOCK_NONBLOCKING, etc. Those are actually all tied together. I also
>>tried
>> building capnproto with -DKJ_USE_EPOLL=0 to get around the signalfd
>> dependency, and that¹s what triggered the pipe2, SOCK_NONBLOCKING, etc.
>> was not declared in this scope compiler errors. It turns out that pipe2,
>> etc. are not available on CentOS-5 either. So, -DKJ_USE_EPOLL=0 or not,
>> the compilation fails due to missing headers or symbols in CentOS-5.
>>
>> I am now trying to create a patch for captnproto to enable the build to
>>go
>> through, but it is messy, as other parts of capnproto rely on some of
>>the
>> related headers. What a mess!
>
>Sorry to hear that :-/.
>
>Unfortunately, there's not really anything the manylinux1 image can do
>to help, because the manylinux1 spec says that manylinux1 wheels must
>work with CentOS 5, so by definition code that requires signalfd
>and/or pipe2 cannot be built into a manylinux1 wheel.
>
>If it helps, pipe2(pipefds, <something>) can be straightforwardly
>replaced by
>
>  pipe(pipefds)
>  fcntl(pipefds[0], F_SETFD, <something>)
>  fcntl(pipefds[1], F_SETFD, <somthing>)
>
>(The pipe2 version is atomic with respect to forks, which is why it
>was added, but if you don't have pipe2 then the fcntl version is
>generally going to be fine outside of some fairly obscure cases.)
>
>Your other option is to lie and cheat :-). Technically you *can* build
>on CentOS 6 and then manually rename your .whl file to claim to be a
>manylinux1 wheel, and PyPI will accept it and pip will install it.
>(auditwheel will scream at you, but nothing forces you to listen to
>auditwheel.) Officially I can't recommend this. Unofficially, if you
>go ahead and do it anyway, then please let us know how it goes,
>because your user's reactions will be a very useful data point for
>designing the manylinux2 spec ;-).
>
>(A "manylinux1" wheel that was built on CentOS 6 should work for
>almost every user; AFAIK the one exception is users on CentOS 5.
>Unfortunately it turns out that we don't have any data on how many
>such users are out there [1], but they're certainly fairly rare...)
>
>-n
>
>[1] https://github.com/pypa/pip/pull/3836
>
>-- 
>Nathaniel J. Smith -- https://vorpus.org


Thanks, Nathaniel, for following up here and also in
https://github.com/pypa/manylinux/issues/75. I think that your suggestion
about using CentOS 6 would involve forking the
https://github.com/pypa/manylinux repo, replacing centos:5.11 with
centos:6 and building a new image, including running
build_scripts/build.sh, right?

Best,
Vitaly