[Wheel-builders] error: signalfd.h not found when building manylinux wheel

[Nathaniel Smith]

On Jul 10, 2016 10:20 AM, "Vitaly Kruglikov" <vkruglikov at numenta.com> wrote:
>
>
>
> On 7/9/16, 11:15 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
>
> >On Fri, Jul 8, 2016 at 10:33 PM, Vitaly Kruglikov
> ><vkruglikov at numenta.com> wrote:
> >>
> >>
> >> On 7/8/16, 10:07 PM, "Nathaniel Smith" <njs at pobox.com> wrote:
> >>
> >>>On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov
> >>><vkruglikov at numenta.com>
> >>>wrote:
> >>>> I am attempting to build a manylinux wheel from the nupic.core
project
> >>>>in
> >>>> github. I am using the docker image quay.io/pypa/manylinux1_x86_64.
> >>>> nupic.core builds and statically links against the capnproto library,
> >>>>which
> >>>> relies on signalfd.h. Unfortunately, the docker image
> >>>> quay.io/pypa/manylinux1_x86_64 does not provide  signalfd.h, so my
> >>>>build
> >>>> fails like this:
> >>>>
> >>>> Linking CXX static library libkj.a
> >>>> [ 27%] Built target kj
> >>>> [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o
> >>>> [ 30%] Building CXX object
> >>>>src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o
> >>>>
>
>>>>/nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix
> >>>>.c
> >>>>++:36:26:
> >>>> fatal error: sys/signalfd.h: No such file or directory
> >>>>  #include <sys/signalfd.h>
> >>>>
> >>>> What is the recommended solution for this problem?
> >>>
> >>>man signalfd says:
> >>>
> >>>VERSIONS
> >>>       signalfd() is available on Linux since kernel 2.6.22.  Working
> >>>support
> >>>       is  provided  in  glibc since version 2.8.  The signalfd4()
> >>>system
> >>>call
> >>>       (see NOTES) is available on Linux since kernel 2.6.27.
> >>>
> >>>CentOS 5 ships glibc 2.5, so signalfd is simply not available to
> >>>manylinux1 wheels. I guess the recommended solution is to modify your
> >>>code so that it doesn't require signalfd? It looks you aren't the only
> >>>person to run into this this week...
> >>>
> >>>    https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU
> >>>
> >>>and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto
> >>>not to use signalfd.
> >>>
> >>>This is an interesting data point for the benefits of defining a
> >>>CentOS-6-based manylinux2, though...
> >>>
> >>>-n
> >>>
> >>>--
> >>>Nathaniel J. Smith -- https://vorpus.org
> >>
> >>
> >> Thanks Nathaniel, unfortunately it is not as simple as that.
> >> Unfortunately, capnproto is not my code, so I am somewhat limited in
> >>what
> >> I can do with it. You also replied to a similar question concerning
> >>pipe2,
> >> SOCK_NONBLOCKING, etc. Those are actually all tied together. I also
> >>tried
> >> building capnproto with -DKJ_USE_EPOLL=0 to get around the signalfd
> >> dependency, and that¹s what triggered the pipe2, SOCK_NONBLOCKING, etc.
> >> was not declared in this scope compiler errors. It turns out that
pipe2,
> >> etc. are not available on CentOS-5 either. So, -DKJ_USE_EPOLL=0 or not,
> >> the compilation fails due to missing headers or symbols in CentOS-5.
> >>
> >> I am now trying to create a patch for captnproto to enable the build to
> >>go
> >> through, but it is messy, as other parts of capnproto rely on some of
> >>the
> >> related headers. What a mess!
> >
> >Sorry to hear that :-/.
> >
> >Unfortunately, there's not really anything the manylinux1 image can do
> >to help, because the manylinux1 spec says that manylinux1 wheels must
> >work with CentOS 5, so by definition code that requires signalfd
> >and/or pipe2 cannot be built into a manylinux1 wheel.
> >
> >If it helps, pipe2(pipefds, <something>) can be straightforwardly
> >replaced by
> >
> >  pipe(pipefds)
> >  fcntl(pipefds[0], F_SETFD, <something>)
> >  fcntl(pipefds[1], F_SETFD, <somthing>)
> >
> >(The pipe2 version is atomic with respect to forks, which is why it
> >was added, but if you don't have pipe2 then the fcntl version is
> >generally going to be fine outside of some fairly obscure cases.)
> >
> >Your other option is to lie and cheat :-). Technically you *can* build
> >on CentOS 6 and then manually rename your .whl file to claim to be a
> >manylinux1 wheel, and PyPI will accept it and pip will install it.
> >(auditwheel will scream at you, but nothing forces you to listen to
> >auditwheel.) Officially I can't recommend this. Unofficially, if you
> >go ahead and do it anyway, then please let us know how it goes,
> >because your user's reactions will be a very useful data point for
> >designing the manylinux2 spec ;-).
> >
> >(A "manylinux1" wheel that was built on CentOS 6 should work for
> >almost every user; AFAIK the one exception is users on CentOS 5.
> >Unfortunately it turns out that we don't have any data on how many
> >such users are out there [1], but they're certainly fairly rare...)
> >
> >-n
> >
> >[1] https://github.com/pypa/pip/pull/3836
> >
> >--
> >Nathaniel J. Smith -- https://vorpus.org
>
>
> Thanks, Nathaniel, for following up here and also in
> https://github.com/pypa/manylinux/issues/75. I think that your suggestion
> about using CentOS 6 would involve forking the
> https://github.com/pypa/manylinux repo, replacing centos:5.11 with
> centos:6 and building a new image, including running
> build_scripts/build.sh, right?

That would be a reasonable approach, yeah. You'll probably find you need to
tweak a few other things, like I think the toolchain repo we're using is
also centos5 specific. Good luck and let us know how it goes!

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20160710/744af2e6/attachment-0001.html>