From matthew.brett at gmail.com Mon Jul 4 16:11:35 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 4 Jul 2016 21:11:35 +0100 Subject: [Wheel-builders] Manylinux, OSX wheel building framework for travis-ci Message-ID: Hi, I've been experimenting with a submodule to support building and testing OSX and manylinux wheels from the same repository, using travis-ci. The README explains the idea : https://github.com/matthew-brett/multibuild#utilities-for-building-on-travis-ci-with-osx-and-linux In brief - it's fairly easy to set up a repository to build your favorite package, by adding a couple of submodules (your package, and the multibuild repo), adapting the template ".travis.yml" file, and adding possibly minimal build / test configuration in a bash file "config.sh". The framework will run build and test on OSX and both 32 and 64-bit Linux. Examples of some repositories using this setup at the end of https://github.com/matthew-brett/multibuild . It appears to be flexible enough for some fairly exotic builds - such as scipy, matplotlib, and Pillow. I'd be very happy for feedback, bug reports, feature requests, Cheers, Matthew From olivier.grisel at ensta.org Tue Jul 5 05:32:57 2016 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Tue, 5 Jul 2016 11:32:57 +0200 Subject: [Wheel-builders] Manylinux, OSX wheel building framework for travis-ci In-Reply-To: References: Message-ID: Interesting. Out of curiosity , why do you use "language: erlang" and "language: objective-c" in the .travis.yml file. I agree that python is not yet supported for os: osx (as per https://github.com/travis-ci/travis-ci/issues/2312) but why would "language: generic" not work? -- Olivier From matthew.brett at gmail.com Tue Jul 5 07:07:26 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 5 Jul 2016 12:07:26 +0100 Subject: [Wheel-builders] Manylinux, OSX wheel building framework for travis-ci In-Reply-To: References: Message-ID: On Tue, Jul 5, 2016 at 10:32 AM, Olivier Grisel wrote: > Interesting. Out of curiosity , why do you use "language: erlang" and > "language: objective-c" in the .travis.yml file. I agree that python > is not yet supported for os: osx (as per > https://github.com/travis-ci/travis-ci/issues/2312) but why would > "language: generic" not work? Sorry, those were all due to habit acquired from the early days of OSX builds - I've changed the language to 'generic' for OSX, seems to work fine. Cheers, Matthew From vkruglikov at numenta.com Fri Jul 8 19:31:28 2016 From: vkruglikov at numenta.com (Vitaly Kruglikov) Date: Fri, 8 Jul 2016 23:31:28 +0000 Subject: [Wheel-builders] Missing newer pipe2, socket, and FD options in manylinux docker image Message-ID: I am using the docker image quay.io/pypa/manylinux1_x86_64 and am running into the compilation issues below, resulting from missing SOCK_NONBLOCK, SOCK_CLOEXEC, pipe2, etc., in this CentOS5-based docker image. What is a reasonable solution? /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++: In member function 'int kj::{anonymous}::SocketAddress::socket(int) const': /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:348:13: error: 'SOCK_NONBLOCK' was not declared in this scope type |= SOCK_NONBLOCK | SOCK_CLOEXEC; ^ /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:348:29: error: 'SOCK_CLOEXEC' was not declared in this scope type |= SOCK_NONBLOCK | SOCK_CLOEXEC; ^ In file included from /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:24:0: /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++: In lambda function: /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:648:38: error: 'O_CLOEXEC' was not declared in this scope KJ_SYSCALL(pipe2(fds, O_NONBLOCK | O_CLOEXEC)); ^ /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:648:47: error: 'pipe2' was not declared in this scope KJ_SYSCALL(pipe2(fds, O_NONBLOCK | O_CLOEXEC)); Many thanks, Vitaly -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jul 8 20:37:11 2016 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 8 Jul 2016 17:37:11 -0700 Subject: [Wheel-builders] Missing newer pipe2, socket, and FD options in manylinux docker image In-Reply-To: References: Message-ID: It's hard to say, since it depends on the details of your build system, but it looks like the code you're trying to build is missing some #include statements -- probably some combination of #include , #include , #include . You might also have to define _GNU_SOURCE to get access to pipe2, via -D_GNU_SOURCE on the command line or #define _GNU_SOURCE in the source code. There are a lot of finicky things that could be causing this to trigger on the manylinux docker image specifically -- maybe your configure script is misbehaving, or maybe on your regular build system there's some coincidence where one of the files that you do #include accidentally pulls in one of these necessary files and thus hides the latent bug, or... -n On Fri, Jul 8, 2016 at 4:31 PM, Vitaly Kruglikov wrote: > I am using the docker image quay.io/pypa/manylinux1_x86_64 and am running > into the compilation issues below, resulting from missing SOCK_NONBLOCK, > SOCK_CLOEXEC, pipe2, etc., in this CentOS5-based docker image. What is a > reasonable solution? > > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++: > In member function ?int kj::{anonymous}::SocketAddress::socket(int) const?: > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:348:13: > error: ?SOCK_NONBLOCK? was not declared in this scope > type |= SOCK_NONBLOCK | SOCK_CLOEXEC; > ^ > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:348:29: > error: ?SOCK_CLOEXEC? was not declared in this scope > type |= SOCK_NONBLOCK | SOCK_CLOEXEC; > ^ > In file included from > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:24:0: > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++: > In lambda function: > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:648:38: > error: ?O_CLOEXEC? was not declared in this scope > KJ_SYSCALL(pipe2(fds, O_NONBLOCK | O_CLOEXEC)); > ^ > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-io.c++:648:47: > error: ?pipe2? was not declared in this scope > KJ_SYSCALL(pipe2(fds, O_NONBLOCK | O_CLOEXEC)); > > > Many thanks, > Vitaly > > > > > _______________________________________________ > Wheel-builders mailing list > Wheel-builders at python.org > https://mail.python.org/mailman/listinfo/wheel-builders > -- Nathaniel J. Smith -- https://vorpus.org From vkruglikov at numenta.com Fri Jul 8 18:23:29 2016 From: vkruglikov at numenta.com (Vitaly Kruglikov) Date: Fri, 8 Jul 2016 22:23:29 +0000 Subject: [Wheel-builders] error: signalfd.h not found when building manylinux wheel Message-ID: I am attempting to build a manylinux wheel from the nupic.core project in github. I am using the docker image quay.io/pypa/manylinux1_x86_64. nupic.core builds and statically links against the capnproto library, which relies on signalfd.h. Unfortunately, the docker image quay.io/pypa/manylinux1_x86_64 does not provide signalfd.h, so my build fails like this: Linking CXX static library libkj.a [ 27%] Built target kj [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o [ 30%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix.c++:36:26: fatal error: sys/signalfd.h: No such file or directory #include What is the recommended solution for this problem? Many thanks, Vitaly -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Jul 9 01:07:51 2016 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 8 Jul 2016 22:07:51 -0700 Subject: [Wheel-builders] error: signalfd.h not found when building manylinux wheel In-Reply-To: References: Message-ID: On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov wrote: > I am attempting to build a manylinux wheel from the nupic.core project in > github. I am using the docker image quay.io/pypa/manylinux1_x86_64. > nupic.core builds and statically links against the capnproto library, which > relies on signalfd.h. Unfortunately, the docker image > quay.io/pypa/manylinux1_x86_64 does not provide signalfd.h, so my build > fails like this: > > Linking CXX static library libkj.a > [ 27%] Built target kj > [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o > [ 30%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o > /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix.c++:36:26: > fatal error: sys/signalfd.h: No such file or directory > #include > > What is the recommended solution for this problem? man signalfd says: VERSIONS signalfd() is available on Linux since kernel 2.6.22. Working support is provided in glibc since version 2.8. The signalfd4() system call (see NOTES) is available on Linux since kernel 2.6.27. CentOS 5 ships glibc 2.5, so signalfd is simply not available to manylinux1 wheels. I guess the recommended solution is to modify your code so that it doesn't require signalfd? It looks you aren't the only person to run into this this week... https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto not to use signalfd. This is an interesting data point for the benefits of defining a CentOS-6-based manylinux2, though... -n -- Nathaniel J. Smith -- https://vorpus.org From vkruglikov at numenta.com Sat Jul 9 01:33:49 2016 From: vkruglikov at numenta.com (Vitaly Kruglikov) Date: Sat, 9 Jul 2016 05:33:49 +0000 Subject: [Wheel-builders] error: signalfd.h not found when building manylinux wheel In-Reply-To: References: Message-ID: On 7/8/16, 10:07 PM, "Nathaniel Smith" wrote: >On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov >wrote: >> I am attempting to build a manylinux wheel from the nupic.core project >>in >> github. I am using the docker image quay.io/pypa/manylinux1_x86_64. >> nupic.core builds and statically links against the capnproto library, >>which >> relies on signalfd.h. Unfortunately, the docker image >> quay.io/pypa/manylinux1_x86_64 does not provide signalfd.h, so my build >> fails like this: >> >> Linking CXX static library libkj.a >> [ 27%] Built target kj >> [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o >> [ 30%] Building CXX object >>src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o >> >>/nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix.c >>++:36:26: >> fatal error: sys/signalfd.h: No such file or directory >> #include >> >> What is the recommended solution for this problem? > >man signalfd says: > >VERSIONS > signalfd() is available on Linux since kernel 2.6.22. Working >support > is provided in glibc since version 2.8. The signalfd4() system >call > (see NOTES) is available on Linux since kernel 2.6.27. > >CentOS 5 ships glibc 2.5, so signalfd is simply not available to >manylinux1 wheels. I guess the recommended solution is to modify your >code so that it doesn't require signalfd? It looks you aren't the only >person to run into this this week... > > https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU > >and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto >not to use signalfd. > >This is an interesting data point for the benefits of defining a >CentOS-6-based manylinux2, though... > >-n > >-- >Nathaniel J. Smith -- https://vorpus.org Thanks Nathaniel, unfortunately it is not as simple as that. Unfortunately, capnproto is not my code, so I am somewhat limited in what I can do with it. You also replied to a similar question concerning pipe2, SOCK_NONBLOCKING, etc. Those are actually all tied together. I also tried building capnproto with -DKJ_USE_EPOLL=0 to get around the signalfd dependency, and that?s what triggered the pipe2, SOCK_NONBLOCKING, etc. was not declared in this scope compiler errors. It turns out that pipe2, etc. are not available on CentOS-5 either. So, -DKJ_USE_EPOLL=0 or not, the compilation fails due to missing headers or symbols in CentOS-5. I am now trying to create a patch for captnproto to enable the build to go through, but it is messy, as other parts of capnproto rely on some of the related headers. What a mess! Best, Vitaly From njs at pobox.com Sun Jul 10 02:15:02 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 9 Jul 2016 23:15:02 -0700 Subject: [Wheel-builders] error: signalfd.h not found when building manylinux wheel In-Reply-To: References: Message-ID: On Fri, Jul 8, 2016 at 10:33 PM, Vitaly Kruglikov wrote: > > > On 7/8/16, 10:07 PM, "Nathaniel Smith" wrote: > >>On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov >>wrote: >>> I am attempting to build a manylinux wheel from the nupic.core project >>>in >>> github. I am using the docker image quay.io/pypa/manylinux1_x86_64. >>> nupic.core builds and statically links against the capnproto library, >>>which >>> relies on signalfd.h. Unfortunately, the docker image >>> quay.io/pypa/manylinux1_x86_64 does not provide signalfd.h, so my build >>> fails like this: >>> >>> Linking CXX static library libkj.a >>> [ 27%] Built target kj >>> [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o >>> [ 30%] Building CXX object >>>src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o >>> >>>/nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix.c >>>++:36:26: >>> fatal error: sys/signalfd.h: No such file or directory >>> #include >>> >>> What is the recommended solution for this problem? >> >>man signalfd says: >> >>VERSIONS >> signalfd() is available on Linux since kernel 2.6.22. Working >>support >> is provided in glibc since version 2.8. The signalfd4() system >>call >> (see NOTES) is available on Linux since kernel 2.6.27. >> >>CentOS 5 ships glibc 2.5, so signalfd is simply not available to >>manylinux1 wheels. I guess the recommended solution is to modify your >>code so that it doesn't require signalfd? It looks you aren't the only >>person to run into this this week... >> >> https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU >> >>and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto >>not to use signalfd. >> >>This is an interesting data point for the benefits of defining a >>CentOS-6-based manylinux2, though... >> >>-n >> >>-- >>Nathaniel J. Smith -- https://vorpus.org > > > Thanks Nathaniel, unfortunately it is not as simple as that. > Unfortunately, capnproto is not my code, so I am somewhat limited in what > I can do with it. You also replied to a similar question concerning pipe2, > SOCK_NONBLOCKING, etc. Those are actually all tied together. I also tried > building capnproto with -DKJ_USE_EPOLL=0 to get around the signalfd > dependency, and that?s what triggered the pipe2, SOCK_NONBLOCKING, etc. > was not declared in this scope compiler errors. It turns out that pipe2, > etc. are not available on CentOS-5 either. So, -DKJ_USE_EPOLL=0 or not, > the compilation fails due to missing headers or symbols in CentOS-5. > > I am now trying to create a patch for captnproto to enable the build to go > through, but it is messy, as other parts of capnproto rely on some of the > related headers. What a mess! Sorry to hear that :-/. Unfortunately, there's not really anything the manylinux1 image can do to help, because the manylinux1 spec says that manylinux1 wheels must work with CentOS 5, so by definition code that requires signalfd and/or pipe2 cannot be built into a manylinux1 wheel. If it helps, pipe2(pipefds, ) can be straightforwardly replaced by pipe(pipefds) fcntl(pipefds[0], F_SETFD, ) fcntl(pipefds[1], F_SETFD, ) (The pipe2 version is atomic with respect to forks, which is why it was added, but if you don't have pipe2 then the fcntl version is generally going to be fine outside of some fairly obscure cases.) Your other option is to lie and cheat :-). Technically you *can* build on CentOS 6 and then manually rename your .whl file to claim to be a manylinux1 wheel, and PyPI will accept it and pip will install it. (auditwheel will scream at you, but nothing forces you to listen to auditwheel.) Officially I can't recommend this. Unofficially, if you go ahead and do it anyway, then please let us know how it goes, because your user's reactions will be a very useful data point for designing the manylinux2 spec ;-). (A "manylinux1" wheel that was built on CentOS 6 should work for almost every user; AFAIK the one exception is users on CentOS 5. Unfortunately it turns out that we don't have any data on how many such users are out there [1], but they're certainly fairly rare...) -n [1] https://github.com/pypa/pip/pull/3836 -- Nathaniel J. Smith -- https://vorpus.org From vkruglikov at numenta.com Sun Jul 10 11:46:31 2016 From: vkruglikov at numenta.com (Vitaly Kruglikov) Date: Sun, 10 Jul 2016 15:46:31 +0000 Subject: [Wheel-builders] error: signalfd.h not found when building manylinux wheel In-Reply-To: References: Message-ID: On 7/9/16, 11:15 PM, "Nathaniel Smith" wrote: >On Fri, Jul 8, 2016 at 10:33 PM, Vitaly Kruglikov > wrote: >> >> >> On 7/8/16, 10:07 PM, "Nathaniel Smith" wrote: >> >>>On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov >>> >>>wrote: >>>> I am attempting to build a manylinux wheel from the nupic.core project >>>>in >>>> github. I am using the docker image quay.io/pypa/manylinux1_x86_64. >>>> nupic.core builds and statically links against the capnproto library, >>>>which >>>> relies on signalfd.h. Unfortunately, the docker image >>>> quay.io/pypa/manylinux1_x86_64 does not provide signalfd.h, so my >>>>build >>>> fails like this: >>>> >>>> Linking CXX static library libkj.a >>>> [ 27%] Built target kj >>>> [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o >>>> [ 30%] Building CXX object >>>>src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o >>>> >>>>/nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix >>>>.c >>>>++:36:26: >>>> fatal error: sys/signalfd.h: No such file or directory >>>> #include >>>> >>>> What is the recommended solution for this problem? >>> >>>man signalfd says: >>> >>>VERSIONS >>> signalfd() is available on Linux since kernel 2.6.22. Working >>>support >>> is provided in glibc since version 2.8. The signalfd4() >>>system >>>call >>> (see NOTES) is available on Linux since kernel 2.6.27. >>> >>>CentOS 5 ships glibc 2.5, so signalfd is simply not available to >>>manylinux1 wheels. I guess the recommended solution is to modify your >>>code so that it doesn't require signalfd? It looks you aren't the only >>>person to run into this this week... >>> >>> https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU >>> >>>and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto >>>not to use signalfd. >>> >>>This is an interesting data point for the benefits of defining a >>>CentOS-6-based manylinux2, though... >>> >>>-n >>> >>>-- >>>Nathaniel J. Smith -- https://vorpus.org >> >> >> Thanks Nathaniel, unfortunately it is not as simple as that. >> Unfortunately, capnproto is not my code, so I am somewhat limited in >>what >> I can do with it. You also replied to a similar question concerning >>pipe2, >> SOCK_NONBLOCKING, etc. Those are actually all tied together. I also >>tried >> building capnproto with -DKJ_USE_EPOLL=0 to get around the signalfd >> dependency, and that?s what triggered the pipe2, SOCK_NONBLOCKING, etc. >> was not declared in this scope compiler errors. It turns out that pipe2, >> etc. are not available on CentOS-5 either. So, -DKJ_USE_EPOLL=0 or not, >> the compilation fails due to missing headers or symbols in CentOS-5. >> >> I am now trying to create a patch for captnproto to enable the build to >>go >> through, but it is messy, as other parts of capnproto rely on some of >>the >> related headers. What a mess! > >Sorry to hear that :-/. > >Unfortunately, there's not really anything the manylinux1 image can do >to help, because the manylinux1 spec says that manylinux1 wheels must >work with CentOS 5, so by definition code that requires signalfd >and/or pipe2 cannot be built into a manylinux1 wheel. > >If it helps, pipe2(pipefds, ) can be straightforwardly >replaced by > > pipe(pipefds) > fcntl(pipefds[0], F_SETFD, ) > fcntl(pipefds[1], F_SETFD, ) > >(The pipe2 version is atomic with respect to forks, which is why it >was added, but if you don't have pipe2 then the fcntl version is >generally going to be fine outside of some fairly obscure cases.) > >Your other option is to lie and cheat :-). Technically you *can* build >on CentOS 6 and then manually rename your .whl file to claim to be a >manylinux1 wheel, and PyPI will accept it and pip will install it. >(auditwheel will scream at you, but nothing forces you to listen to >auditwheel.) Officially I can't recommend this. Unofficially, if you >go ahead and do it anyway, then please let us know how it goes, >because your user's reactions will be a very useful data point for >designing the manylinux2 spec ;-). > >(A "manylinux1" wheel that was built on CentOS 6 should work for >almost every user; AFAIK the one exception is users on CentOS 5. >Unfortunately it turns out that we don't have any data on how many >such users are out there [1], but they're certainly fairly rare...) > >-n > >[1] https://github.com/pypa/pip/pull/3836 > >-- >Nathaniel J. Smith -- https://vorpus.org Thanks, Nathaniel, for following up here and also in https://github.com/pypa/manylinux/issues/75. I think that your suggestion about using CentOS 6 would involve forking the https://github.com/pypa/manylinux repo, replacing centos:5.11 with centos:6 and building a new image, including running build_scripts/build.sh, right? Best, Vitaly From njs at pobox.com Sun Jul 10 14:08:31 2016 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 10 Jul 2016 11:08:31 -0700 Subject: [Wheel-builders] error: signalfd.h not found when building manylinux wheel In-Reply-To: References: Message-ID: On Jul 10, 2016 10:20 AM, "Vitaly Kruglikov" wrote: > > > > On 7/9/16, 11:15 PM, "Nathaniel Smith" wrote: > > >On Fri, Jul 8, 2016 at 10:33 PM, Vitaly Kruglikov > > wrote: > >> > >> > >> On 7/8/16, 10:07 PM, "Nathaniel Smith" wrote: > >> > >>>On Fri, Jul 8, 2016 at 3:23 PM, Vitaly Kruglikov > >>> > >>>wrote: > >>>> I am attempting to build a manylinux wheel from the nupic.core project > >>>>in > >>>> github. I am using the docker image quay.io/pypa/manylinux1_x86_64. > >>>> nupic.core builds and statically links against the capnproto library, > >>>>which > >>>> relies on signalfd.h. Unfortunately, the docker image > >>>> quay.io/pypa/manylinux1_x86_64 does not provide signalfd.h, so my > >>>>build > >>>> fails like this: > >>>> > >>>> Linking CXX static library libkj.a > >>>> [ 27%] Built target kj > >>>> [ 29%] Building CXX object src/kj/CMakeFiles/kj-async.dir/async.c++.o > >>>> [ 30%] Building CXX object > >>>>src/kj/CMakeFiles/kj-async.dir/async-unix.c++.o > >>>> > >>>>/nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/async-unix > >>>>.c > >>>>++:36:26: > >>>> fatal error: sys/signalfd.h: No such file or directory > >>>> #include > >>>> > >>>> What is the recommended solution for this problem? > >>> > >>>man signalfd says: > >>> > >>>VERSIONS > >>> signalfd() is available on Linux since kernel 2.6.22. Working > >>>support > >>> is provided in glibc since version 2.8. The signalfd4() > >>>system > >>>call > >>> (see NOTES) is available on Linux since kernel 2.6.27. > >>> > >>>CentOS 5 ships glibc 2.5, so signalfd is simply not available to > >>>manylinux1 wheels. I guess the recommended solution is to modify your > >>>code so that it doesn't require signalfd? It looks you aren't the only > >>>person to run into this this week... > >>> > >>> https://groups.google.com/forum/#!topic/capnproto/OpH9RtOBcZU > >>> > >>>and the suggestion there is to use -DKJ_USE_EPOLL=0 to tell capnproto > >>>not to use signalfd. > >>> > >>>This is an interesting data point for the benefits of defining a > >>>CentOS-6-based manylinux2, though... > >>> > >>>-n > >>> > >>>-- > >>>Nathaniel J. Smith -- https://vorpus.org > >> > >> > >> Thanks Nathaniel, unfortunately it is not as simple as that. > >> Unfortunately, capnproto is not my code, so I am somewhat limited in > >>what > >> I can do with it. You also replied to a similar question concerning > >>pipe2, > >> SOCK_NONBLOCKING, etc. Those are actually all tied together. I also > >>tried > >> building capnproto with -DKJ_USE_EPOLL=0 to get around the signalfd > >> dependency, and that?s what triggered the pipe2, SOCK_NONBLOCKING, etc. > >> was not declared in this scope compiler errors. It turns out that pipe2, > >> etc. are not available on CentOS-5 either. So, -DKJ_USE_EPOLL=0 or not, > >> the compilation fails due to missing headers or symbols in CentOS-5. > >> > >> I am now trying to create a patch for captnproto to enable the build to > >>go > >> through, but it is messy, as other parts of capnproto rely on some of > >>the > >> related headers. What a mess! > > > >Sorry to hear that :-/. > > > >Unfortunately, there's not really anything the manylinux1 image can do > >to help, because the manylinux1 spec says that manylinux1 wheels must > >work with CentOS 5, so by definition code that requires signalfd > >and/or pipe2 cannot be built into a manylinux1 wheel. > > > >If it helps, pipe2(pipefds, ) can be straightforwardly > >replaced by > > > > pipe(pipefds) > > fcntl(pipefds[0], F_SETFD, ) > > fcntl(pipefds[1], F_SETFD, ) > > > >(The pipe2 version is atomic with respect to forks, which is why it > >was added, but if you don't have pipe2 then the fcntl version is > >generally going to be fine outside of some fairly obscure cases.) > > > >Your other option is to lie and cheat :-). Technically you *can* build > >on CentOS 6 and then manually rename your .whl file to claim to be a > >manylinux1 wheel, and PyPI will accept it and pip will install it. > >(auditwheel will scream at you, but nothing forces you to listen to > >auditwheel.) Officially I can't recommend this. Unofficially, if you > >go ahead and do it anyway, then please let us know how it goes, > >because your user's reactions will be a very useful data point for > >designing the manylinux2 spec ;-). > > > >(A "manylinux1" wheel that was built on CentOS 6 should work for > >almost every user; AFAIK the one exception is users on CentOS 5. > >Unfortunately it turns out that we don't have any data on how many > >such users are out there [1], but they're certainly fairly rare...) > > > >-n > > > >[1] https://github.com/pypa/pip/pull/3836 > > > >-- > >Nathaniel J. Smith -- https://vorpus.org > > > Thanks, Nathaniel, for following up here and also in > https://github.com/pypa/manylinux/issues/75. I think that your suggestion > about using CentOS 6 would involve forking the > https://github.com/pypa/manylinux repo, replacing centos:5.11 with > centos:6 and building a new image, including running > build_scripts/build.sh, right? That would be a reasonable approach, yeah. You'll probably find you need to tweak a few other things, like I think the toolchain repo we're using is also centos5 specific. Good luck and let us know how it goes! -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Jul 13 20:11:52 2016 From: dholth at gmail.com (Daniel Holth) Date: Thu, 14 Jul 2016 00:11:52 +0000 Subject: [Wheel-builders] Enscons Message-ID: I've been fishing for feedback or about enscons: https://bitbucket.org/dholth/enscons/src It adds wheel building, extension compiling and setup.py emulation to SCons: packaging without distutils. Still very early. Thanks, Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: From amitsaha.in at gmail.com Thu Jul 14 09:02:07 2016 From: amitsaha.in at gmail.com (Amit Saha) Date: Thu, 14 Jul 2016 23:02:07 +1000 Subject: [Wheel-builders] Query regarding allowed shared libraries Message-ID: Hello, I was reading https://www.python.org/dev/peps/pep-0513/ and noticed the list of shared libraries a C extension is allowed to link against. If, say I were to create a manylinux1 wheel for librabbitmq, would I have to then statically link to the librabbitmq C library? And similarly for a Python binding to any C library? Best Wishes, Amit. -- http://echorand.me From rmcgibbo at gmail.com Thu Jul 14 09:26:03 2016 From: rmcgibbo at gmail.com (rmcgibbo at gmail.com) Date: Thu, 14 Jul 2016 09:26:03 -0400 Subject: [Wheel-builders] Query regarding allowed shared libraries In-Reply-To: References: Message-ID: Hi, You have two options. One is to statically link. The other, which is often easier, is to dynamically link and bundle the required shared libraries inside the wheel. The auditwheel tool can walk the linker tree to locate these libraries for you and graft them into the wheel. -Robert Sent from my iPhone > On Jul 14, 2016, at 9:02 AM, Amit Saha wrote: > > Hello, > > I was reading https://www.python.org/dev/peps/pep-0513/ and noticed > the list of shared libraries a C extension is allowed to link against. > If, say I were to create a manylinux1 wheel for librabbitmq, would I > have to then statically link to the librabbitmq C library? And > similarly for a Python binding to any C library? > > Best Wishes, > Amit. > > -- > http://echorand.me > _______________________________________________ > Wheel-builders mailing list > Wheel-builders at python.org > https://mail.python.org/mailman/listinfo/wheel-builders From amitsaha.in at gmail.com Sun Jul 17 08:20:09 2016 From: amitsaha.in at gmail.com (Amit Saha) Date: Sun, 17 Jul 2016 22:20:09 +1000 Subject: [Wheel-builders] Query regarding allowed shared libraries In-Reply-To: References: Message-ID: On Thu, Jul 14, 2016 at 11:26 PM, wrote: > Hi, > > You have two options. One is to statically link. The other, which is often easier, is to dynamically link and bundle the required shared libraries inside the wheel. The auditwheel tool can walk the linker tree to locate these libraries for you and graft them into the wheel. Thanks for the reply. I am currently trying to experiment with building manylinux1 wheels for https://pypi.python.org/pypi/librabbitmq - I will share how it goes. > > -Robert > > Sent from my iPhone > >> On Jul 14, 2016, at 9:02 AM, Amit Saha wrote: >> >> Hello, >> >> I was reading https://www.python.org/dev/peps/pep-0513/ and noticed >> the list of shared libraries a C extension is allowed to link against. >> If, say I were to create a manylinux1 wheel for librabbitmq, would I >> have to then statically link to the librabbitmq C library? And >> similarly for a Python binding to any C library? >> >> Best Wishes, >> Amit. >> >> -- >> http://echorand.me >> _______________________________________________ >> Wheel-builders mailing list >> Wheel-builders at python.org >> https://mail.python.org/mailman/listinfo/wheel-builders -- http://echorand.me From matthew.brett at gmail.com Mon Jul 18 16:43:55 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 18 Jul 2016 21:43:55 +0100 Subject: [Wheel-builders] Pre-release Python on docker image? Message-ID: Hi, New Python versions are a real problem for our current wheel system. When Python 3.5 came out, suddenly the default Python installed from Python.org did not have any binary wheels ready for it, so an install of scipy on OSX went from a few seconds to some very long time or a crash if you didn't have the requirements. In order to avoid that, we'll need to make it easy to build and ship wheels for upcoming Python releases. I just implemented that for OSX / travis-ci over at https://github.com/matthew-brett/multibuild - you can add e.g. Python version string `3.6.0a3` to build against, which downloads and installs the pre-release installer for OSX. What to do about manylinux? I guess we need to have a pre-release Python built on the docker image. Cheers, Matthew From njs at pobox.com Mon Jul 18 18:58:46 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 18 Jul 2016 17:58:46 -0500 Subject: [Wheel-builders] Pre-release Python on docker image? In-Reply-To: References: Message-ID: On Mon, Jul 18, 2016 at 3:43 PM, Matthew Brett wrote: > Hi, > > New Python versions are a real problem for our current wheel system. > When Python 3.5 came out, suddenly the default Python installed from > Python.org did not have any binary wheels ready for it, so an install > of scipy on OSX went from a few seconds to some very long time or a > crash if you didn't have the requirements. > > In order to avoid that, we'll need to make it easy to build and ship > wheels for upcoming Python releases. > > I just implemented that for OSX / travis-ci over at > https://github.com/matthew-brett/multibuild - you can add e.g. Python > version string `3.6.0a3` to build against, which downloads and > installs the pre-release installer for OSX. > > What to do about manylinux? I guess we need to have a pre-release > Python built on the docker image. The problem here is that the 3.6 ABI will not be finalized until 3.6-final is actually released -- any wheels built on 3.6-prereleases could potentially segfault or whatever with the 3.6-final release. Hopefully this won't happen very often in practice, esp. for the later prereleases like the actual release candidates, but in principle it could happen. (For 3.5, the "final rc" was rc3... until they found a nasty problem with how they were building extension modules on Windows, which forced a last minute rc4 [1].) I agree that it would be really good to improve the UX for wheels on new Python releases, but I think it will require some discussion with the Python release managers (and possibly distutils-sig, in case we need finer grained Python version tags than just "3.6"). Definitely no-one should be uploading 3.6.0a3 wheels to PyPI :-) -n [1] http://stevedower.id.au/blog/building-for-python-3-5-part-two/ -- Nathaniel J. Smith -- https://vorpus.org From matthew.brett at gmail.com Mon Jul 18 19:04:24 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 19 Jul 2016 00:04:24 +0100 Subject: [Wheel-builders] Pre-release Python on docker image? In-Reply-To: References: Message-ID: On Mon, Jul 18, 2016 at 11:58 PM, Nathaniel Smith wrote: > On Mon, Jul 18, 2016 at 3:43 PM, Matthew Brett wrote: >> Hi, >> >> New Python versions are a real problem for our current wheel system. >> When Python 3.5 came out, suddenly the default Python installed from >> Python.org did not have any binary wheels ready for it, so an install >> of scipy on OSX went from a few seconds to some very long time or a >> crash if you didn't have the requirements. >> >> In order to avoid that, we'll need to make it easy to build and ship >> wheels for upcoming Python releases. >> >> I just implemented that for OSX / travis-ci over at >> https://github.com/matthew-brett/multibuild - you can add e.g. Python >> version string `3.6.0a3` to build against, which downloads and >> installs the pre-release installer for OSX. >> >> What to do about manylinux? I guess we need to have a pre-release >> Python built on the docker image. > > The problem here is that the 3.6 ABI will not be finalized until > 3.6-final is actually released -- any wheels built on 3.6-prereleases > could potentially segfault or whatever with the 3.6-final release. > Hopefully this won't happen very often in practice, esp. for the later > prereleases like the actual release candidates, but in principle it > could happen. (For 3.5, the "final rc" was rc3... until they found a > nasty problem with how they were building extension modules on > Windows, which forced a last minute rc4 [1].) > > I agree that it would be really good to improve the UX for wheels on > new Python releases, but I think it will require some discussion with > the Python release managers (and possibly distutils-sig, in case we > need finer grained Python version tags than just "3.6"). > > Definitely no-one should be uploading 3.6.0a3 wheels to PyPI :-) OK - good to know. I think we do need a solution of some sort. I'm guessing that people are starting to rely on wheels already, and they aren't going to be happy when everything breaks down for a few weeks after a new Python release. Cheers, Matthew From nathan12343 at gmail.com Thu Jul 21 13:11:19 2016 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Thu, 21 Jul 2016 12:11:19 -0500 Subject: [Wheel-builders] Building wheels for projects that depend on NumPy? Message-ID: Hi all, I'm curious which NumPy version we should use when generating wheels. In particular, will it be a problem if our dependencies use a different NumPy version to build *their* wheels. It's my understanding that the NumPy ABI is forward compatible, so I guess we want to build against a "sufficiently old" version? Thanks for your advice, Nathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jul 21 18:43:36 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 21 Jul 2016 23:43:36 +0100 Subject: [Wheel-builders] Building wheels for projects that depend on NumPy? In-Reply-To: References: Message-ID: Hi, On Thu, Jul 21, 2016 at 6:11 PM, Nathan Goldbaum wrote: > Hi all, > > I'm curious which NumPy version we should use when generating wheels. In > particular, will it be a problem if our dependencies use a different NumPy > version to build *their* wheels. > > It's my understanding that the NumPy ABI is forward compatible, so I guess > we want to build against a "sufficiently old" version? Right - build against the oldest version the project is compatible with : http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312 Cheers, Matthew From njs at pobox.com Thu Jul 21 20:12:22 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 21 Jul 2016 17:12:22 -0700 Subject: [Wheel-builders] Building wheels for projects that depend on NumPy? In-Reply-To: References: Message-ID: Yeah, if you build against numpy version X, and some other package builds against numpy version Y, then your packages will both with with numpy version Z if Z >= X and Z >= X (This is speaking specifically about extensions that use the numpy C api of course -- for packages that just use the numpy python api then it doesn't matter what version you build against.) What you should also do, but currently most people don't do and we need better tooling for, is to make sure that if you build against numpy version X then your wheel should have a versioned dependency on numpy >= X. -n On Jul 21, 2016 3:44 PM, "Matthew Brett" wrote: > Hi, > > On Thu, Jul 21, 2016 at 6:11 PM, Nathan Goldbaum > wrote: > > Hi all, > > > > I'm curious which NumPy version we should use when generating wheels. In > > particular, will it be a problem if our dependencies use a different > NumPy > > version to build *their* wheels. > > > > It's my understanding that the NumPy ABI is forward compatible, so I > guess > > we want to build against a "sufficiently old" version? > > Right - build against the oldest version the project is compatible > with : > http://stackoverflow.com/questions/17709641/valueerror-numpy-dtype-has-the-wrong-size-try-recompiling/18369312#18369312 > > Cheers, > > Matthew > _______________________________________________ > Wheel-builders mailing list > Wheel-builders at python.org > https://mail.python.org/mailman/listinfo/wheel-builders > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin at reportlab.com Fri Jul 22 09:04:49 2016 From: robin at reportlab.com (Robin Becker) Date: Fri, 22 Jul 2016 14:04:49 +0100 Subject: [Wheel-builders] old versions of tiff Message-ID: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> Hi, I've had a request to create manylinux wheels for reportlab. I'm quite willing to do so, but lack experience with travis(we have jenkins running OK). So I started with the code here https://github.com/matthew-brett/manylinux-builds but have immediately run into a problem in that the particular place where tiff is obtained from no longer has the ancient version 2.6.3. I followed some of the discussions about manylinux, but is there a collected set of ancient versions somewhere? -- Robin Becker From matthew.brett at gmail.com Fri Jul 22 09:23:11 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 22 Jul 2016 14:23:11 +0100 Subject: [Wheel-builders] old versions of tiff In-Reply-To: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> Message-ID: Hi, On Fri, Jul 22, 2016 at 2:04 PM, Robin Becker wrote: > Hi, > > I've had a request to create manylinux wheels for reportlab. I'm quite > willing to do so, but lack experience with travis(we have jenkins running > OK). > > So I started with the code here > > https://github.com/matthew-brett/manylinux-builds I'd recommend https://github.com/matthew-brett/multibuild as the first place to start now. manylinux-builds more or less a first draft, where multibuild is the second. > but have immediately run into a problem in that the particular place where > tiff is obtained from no longer has the ancient version 2.6.3. Wow - that is remarkably old. Are you sure that is not a freetype version? Even the docker image, for CentOS 5, installs libtiff 3.8.2, and the oldest tiff download at the tiff website appears to be 3.7.0, which has files dated 2004 at latest. Best, Matthew From robin at reportlab.com Fri Jul 22 09:27:27 2016 From: robin at reportlab.com (Robin Becker) Date: Fri, 22 Jul 2016 14:27:27 +0100 Subject: [Wheel-builders] old versions of tiff In-Reply-To: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> Message-ID: <4ca8ed09-96cd-7ec7-4faa-c7dcf88b31ba@chamonix.reportlab.co.uk> On 22/07/2016 14:04, Robin Becker wrote: > Hi, > > I've had a request to create manylinux wheels for reportlab. I'm quite willing > to do so, but lack experience with travis(we have jenkins running OK). > > So I started with the code here > > https://github.com/matthew-brett/manylinux-builds > > but have immediately run into a problem in that the particular place where tiff > is obtained from no longer has the ancient version 2.6.3. > > I followed some of the discussions about manylinux, but is there a collected set > of ancient versions somewhere? In fact this seems to be a simple bug in the above code ie library_builders.sh contains these definitions at the the top > BZIP2_VERSION="${BZIP2_VERSION:-1.0.6}" > FREETYPE_VERSION="${FREETYPE_VERSION:-2.6.3}" > TIFF_VERSION="${FREETYPE_VERSION:-4.0.6}" > OPENJPEG_VERSION="${OPENJPEG_VERSION:-2.1}" > LCMS2_VERSION="${LCMS2_VERSION:-2.7}" so in the absence of an envirnment variable TIFF gets an older version than is reasonable derived from FREETYPE_VERSION. -- Robin Becker From robin at reportlab.com Fri Jul 22 09:30:21 2016 From: robin at reportlab.com (Robin Becker) Date: Fri, 22 Jul 2016 14:30:21 +0100 Subject: [Wheel-builders] old versions of tiff In-Reply-To: References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> Message-ID: On 22/07/2016 14:23, Matthew Brett wrote: > Hi, > > On Fri, Jul 22, 2016 at 2:04 PM, Robin Becker wrote: >> Hi, >> >> I've had a request to create manylinux wheels for reportlab. I'm quite >> willing to do so, but lack experience with travis(we have jenkins running >> OK). >> >> So I started with the code here >> >> https://github.com/matthew-brett/manylinux-builds > > I'd recommend https://github.com/matthew-brett/multibuild as the first > place to start now. manylinux-builds more or less a first draft, > where multibuild is the second. > OK I'll check that stuff out. >> but have immediately run into a problem in that the particular place where >> tiff is obtained from no longer has the ancient version 2.6.3. > > Wow - that is remarkably old. Are you sure that is not a freetype > version? Even the docker image, for CentOS 5, installs libtiff > 3.8.2, and the oldest tiff download at the tiff website appears to be > 3.7.0, which has files dated 2004 at latest. .... I replied already to the list that it's just a definition issue related to how defaults get set up in library_builders.sh; fixing seems to get me further. -- Robin Becker From matthew.brett at gmail.com Fri Jul 22 09:49:21 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 22 Jul 2016 14:49:21 +0100 Subject: [Wheel-builders] old versions of tiff In-Reply-To: <4ca8ed09-96cd-7ec7-4faa-c7dcf88b31ba@chamonix.reportlab.co.uk> References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> <4ca8ed09-96cd-7ec7-4faa-c7dcf88b31ba@chamonix.reportlab.co.uk> Message-ID: Hi, On Fri, Jul 22, 2016 at 2:27 PM, Robin Becker wrote: > On 22/07/2016 14:04, Robin Becker wrote: >> >> Hi, >> >> I've had a request to create manylinux wheels for reportlab. I'm quite >> willing >> to do so, but lack experience with travis(we have jenkins running OK). >> >> So I started with the code here >> >> https://github.com/matthew-brett/manylinux-builds >> >> but have immediately run into a problem in that the particular place where >> tiff >> is obtained from no longer has the ancient version 2.6.3. >> >> I followed some of the discussions about manylinux, but is there a >> collected set >> of ancient versions somewhere? > > In fact this seems to be a simple bug in the above code ie > library_builders.sh contains these definitions at the the top > >> BZIP2_VERSION="${BZIP2_VERSION:-1.0.6}" >> FREETYPE_VERSION="${FREETYPE_VERSION:-2.6.3}" >> TIFF_VERSION="${FREETYPE_VERSION:-4.0.6}" >> OPENJPEG_VERSION="${OPENJPEG_VERSION:-2.1}" >> LCMS2_VERSION="${LCMS2_VERSION:-2.7}" > > > so in the absence of an envirnment variable TIFF gets an older version than > is reasonable derived from FREETYPE_VERSION. Ah yes - thank you - I had fixed that in multibuilds, but forgot to fix that in manylinux-builds. Best, Matthew From annaraven at gmail.com Fri Jul 22 13:25:56 2016 From: annaraven at gmail.com (Anna Ravenscroft) Date: Fri, 22 Jul 2016 10:25:56 -0700 Subject: [Wheel-builders] old versions of tiff In-Reply-To: References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> Message-ID: So, should we not be using manylinux anymore but instead use multibuild? On Fri, Jul 22, 2016 at 6:23 AM, Matthew Brett wrote: > Hi, > > On Fri, Jul 22, 2016 at 2:04 PM, Robin Becker wrote: > > Hi, > > > > I've had a request to create manylinux wheels for reportlab. I'm quite > > willing to do so, but lack experience with travis(we have jenkins running > > OK). > > > > So I started with the code here > > > > https://github.com/matthew-brett/manylinux-builds > > I'd recommend https://github.com/matthew-brett/multibuild as the first > place to start now. manylinux-builds more or less a first draft, > where multibuild is the second. > > > but have immediately run into a problem in that the particular place > where > > tiff is obtained from no longer has the ancient version 2.6.3. > > Wow - that is remarkably old. Are you sure that is not a freetype > version? Even the docker image, for CentOS 5, installs libtiff > 3.8.2, and the oldest tiff download at the tiff website appears to be > 3.7.0, which has files dated 2004 at latest. > > Best, > > Matthew > _______________________________________________ > Wheel-builders mailing list > Wheel-builders at python.org > https://mail.python.org/mailman/listinfo/wheel-builders > -- cordially, Anna -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jul 22 18:43:36 2016 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 22 Jul 2016 15:43:36 -0700 Subject: [Wheel-builders] old versions of tiff In-Reply-To: References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> Message-ID: If I understand correctly, "manylinux-builds" is the name of Matthew's first set of utilities to help with building manylinux wheels, and "multibuild" is the name of Matthew's second set of utilities to help with building manylinux (and other) wheels. I believe that multibuild is still using the manylinux docker image to build wheels that are compliant with the manylinux spec, and you're still free to build manylinux wheels any way you want as long as they comply with the PEP. But Matthew generally know what he's doing ;-), so if you're trying to figure out the best tooling to build your own wheels (esp. if you're like Matthew and trying to do this for lots of projects at once using free services like Travis-ci), then multibuild is probably worth checking out. Hopefully that clarifies things? -n On Jul 22, 2016 10:26 AM, "Anna Ravenscroft" wrote: > So, should we not be using manylinux anymore but instead use multibuild? > > On Fri, Jul 22, 2016 at 6:23 AM, Matthew Brett > wrote: > >> Hi, >> >> On Fri, Jul 22, 2016 at 2:04 PM, Robin Becker >> wrote: >> > Hi, >> > >> > I've had a request to create manylinux wheels for reportlab. I'm quite >> > willing to do so, but lack experience with travis(we have jenkins >> running >> > OK). >> > >> > So I started with the code here >> > >> > https://github.com/matthew-brett/manylinux-builds >> >> I'd recommend https://github.com/matthew-brett/multibuild as the first >> place to start now. manylinux-builds more or less a first draft, >> where multibuild is the second. >> >> > but have immediately run into a problem in that the particular place >> where >> > tiff is obtained from no longer has the ancient version 2.6.3. >> >> Wow - that is remarkably old. Are you sure that is not a freetype >> version? Even the docker image, for CentOS 5, installs libtiff >> 3.8.2, and the oldest tiff download at the tiff website appears to be >> 3.7.0, which has files dated 2004 at latest. >> >> Best, >> >> Matthew >> _______________________________________________ >> Wheel-builders mailing list >> Wheel-builders at python.org >> https://mail.python.org/mailman/listinfo/wheel-builders >> > > > > -- > cordially, > Anna > > _______________________________________________ > Wheel-builders mailing list > Wheel-builders at python.org > https://mail.python.org/mailman/listinfo/wheel-builders > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robin at reportlab.com Sat Jul 23 03:29:29 2016 From: robin at reportlab.com (Robin Becker) Date: Sat, 23 Jul 2016 08:29:29 +0100 Subject: [Wheel-builders] old versions of tiff In-Reply-To: References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> <4ca8ed09-96cd-7ec7-4faa-c7dcf88b31ba@chamonix.reportlab.co.uk> Message-ID: On 22 July 2016 at 14:49, Matthew Brett wrote: > Hi, > > On Fri, Jul 22, 2016 at 2:27 PM, Robin Becker wrote: >> On 22/07/2016 14:04, Robin Becker wrote: >>> >........ >> is reasonable derived from FREETYPE_VERSION. > > Ah yes - thank you - I had fixed that in multibuilds, but forgot to > fix that in manylinux-builds. > > Best, > > Matthew I have managed to make the multibuild code work for reportlab, but I used the build_pillows.sh script as a start. Took a few goes to make this work properly and an external script to run bothe 32 and 64 bit dockers. I notice that the manylinux wheels are much larger than their windows equivalents eg reportlab-3.3.15-cp27-cp27m-manylinux1_x86_64.whl 2.1M reportlab-3.3.15-cp27-none-win_amd64.whl I assume that's because in the windows builds I am careful to statically link the bits if the freetype library that I use and the manylinux wheels seem to have the whole library present. The linux build process is not trying to link statically, but I assume I could improve that. -- Robin Becker From robin at reportlab.com Sat Jul 23 03:32:12 2016 From: robin at reportlab.com (Robin Becker) Date: Sat, 23 Jul 2016 08:32:12 +0100 Subject: [Wheel-builders] old versions of tiff In-Reply-To: References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> <4ca8ed09-96cd-7ec7-4faa-c7dcf88b31ba@chamonix.reportlab.co.uk> Message-ID: > reportlab-3.3.15-cp27-cp27m-manylinux1_x86_64.whl > 2.1M reportlab-3.3.15-cp27-none-win_amd64.whl missed off the size for the manylinux it was 3.6M -- Robin Becker From matthew.brett at gmail.com Sat Jul 23 06:14:02 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat, 23 Jul 2016 11:14:02 +0100 Subject: [Wheel-builders] old versions of tiff In-Reply-To: References: <0a919b63-712b-ee4c-9a54-a5fd78728daa@chamonix.reportlab.co.uk> Message-ID: On Fri, Jul 22, 2016 at 11:43 PM, Nathaniel Smith wrote: > If I understand correctly, "manylinux-builds" is the name of Matthew's first > set of utilities to help with building manylinux wheels, and "multibuild" is > the name of Matthew's second set of utilities to help with building > manylinux (and other) wheels. Right - manylinux-builds was a set of scripts to generate wheels for a range of projects, using the manylinux docker image. multibuild is a refactored version of the general build utilities in manylinux-builds, designed to be used as a submodule, that has a bash framework for building and testing both OSX and manylinux wheels, again with the manylinux docker image (for building). Cheers, Matthew From robin at reportlab.com Mon Jul 25 10:22:48 2016 From: robin at reportlab.com (Robin Becker) Date: Mon, 25 Jul 2016 15:22:48 +0100 Subject: [Wheel-builders] static linking Message-ID: I'm trying to reduce the size of the reportlab manylinux builds. and have a couple of problems. 1) I don't know how to detect a manylinux platform in my setup.py. A simple solution is to use an evironment variable to change behaviour, but I assume there is to be some official os name or platform. Currently I see something like this > platform=Linux-3.16.0-50-generic-x86_64-with-redhat-5.11-Final > sys.platform=linux2 > os.name=posix 2) Even when I do change behaviour to try and get a static linkage of say freetype agains my code I'm getting linkage errors. Static linkage is said to be OK in pep 513, but I don't know how to get the multi build code to do it. I have checked and there are .a files being produced, but it seems they have the wrong compile time flags my error looks like > /opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: /usr/local/lib/libfreetype.a(ftinit.o): relocation R_X86_64_32 against `tt_driver_class' can not be used when making a shared object; recompile with -fPIC I suppose I need to create a library of the .o files created for the shared build. Then I can link against that .a rather than the non shared one. -- Robin Becker From vkruglikov at numenta.com Mon Jul 25 14:12:45 2016 From: vkruglikov at numenta.com (Vitaly Kruglikov) Date: Mon, 25 Jul 2016 18:12:45 +0000 Subject: [Wheel-builders] manylinux: futex lock in capnproto hangs when running manylinux wheel on Ubuntu 16.04 Message-ID: When I build a manylinux wheel for nupuc.core (https://github.com/numenta/nupic.core/pull/1001), all nupic.core and nupic (https://github.com/numenta/nupic) tests pass on Ubuntu 14.04. However, when I run nupic unit tests on Ubuntu 16.04, I always get a futex lock hang at https://github.com/sandstorm-io/capnproto/blob/v0.5.3/c%2B%2B/src/kj/mutex. c%2B%2B#L87 (a statically-linked copy of capnproto embedded in the python extension .so that?s part of the nupic.bindings manylinux wheel built by nupic.core). The extension build uses shared libs: libc.so.6, libstdc++.so.6, and libgcc_s.so.1. Built and running against Python 2.7.11. I use a custom manylinux docker image that?s created from a fork of manylinux that replaces centos5 with centos6.8 (https://github.com/numenta/manylinux/pull/1) as suggested in https://mail.python.org/pipermail/wheel-builders/2016-July/000175.html. This image has been pushed to quay.io/numenta/manylinux1_x86_64_centos6. The traceback to the hang looks like this: (gdb) bt #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007f2e042d7d77 in kj::_::Mutex::lock (this=0x42b6610, exclusivity=) at /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/mutex.c++:87 #2 0x00007f2e042a658e in kj::MutexGuarded >::lockExclusive (this=0x42b6610) at /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/kj/mutex.h:300 #3 capnp::SchemaLoader::loadNative (this=0x42b6610, nativeSchema=0x7f2e045c1f40 ) at /nupic.core/build/scripts/ThirdParty/Source/CapnProto/src/capnp/schema-load er.c++:2069 #4 0x00007f2e04074761 in capnp::SchemaLoader::loadCompiledTypeAndDependencies (this=) at /nupic.core/build/scripts/ThirdParty/Install/include/capnp/schema-loader.h: 168 #5 capnp::SchemaParser::loadCompiledTypeAndDependencies (this=) at /nupic.core/build/scripts/ThirdParty/Install/include/capnp/schema-parser.h: 83 #6 nupic::getBuilder (pyBuilder=0x7f2e0a0a55f0) at /nupic.core/src/nupic/py_support/PyCapnp.hpp:77 #7 0x00007f2e03fcacd3 in nupic_Network_write__SWIG_2 (self=0x3253090, pyBuilder=) at /nupic.core/build/scripts/src/nupic/bindings/engine_internalPYTHON_wrap.cxx :5287 #8 0x00007f2e03ff878f in _wrap_Network_write__SWIG_2 (nobjs=nobjs at entry=2, swig_obj=swig_obj at entry=0x7ffc743cf0d0) at /nupic.core/build/scripts/src/nupic/bindings/engine_internalPYTHON_wrap.cxx :27690 #9 0x00007f2e03ff8c05 in _wrap_Network_write (self=0x0, args=) at /nupic.core/build/scripts/src/nupic/bindings/engine_internalPYTHON_wrap.cxx :27812 #10 0x00000000004cb26d in PyEval_EvalFrameEx () #11 0x00000000004c22e5 in PyEval_EvalCodeEx () I am going to put in additional effort to isolate the issue to a small code footprint from the vast body of code that it?s in now. However, in the meantime, I was hoping that someone might have run into something similar and might share some helpful clues about the issue or possibly how to debug it efficiently. Many thanks, Vitaly From njs at pobox.com Mon Jul 25 20:24:00 2016 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 25 Jul 2016 17:24:00 -0700 Subject: [Wheel-builders] static linking In-Reply-To: References: Message-ID: On Jul 25, 2016 7:22 AM, "Robin Becker" wrote: > > I'm trying to reduce the size of the reportlab manylinux builds. and have a couple of problems. > > 1) I don't know how to detect a manylinux platform in my setup.py. A simple solution is to use an evironment variable to change behaviour, but I assume there is to be some official os name or platform. Currently I see something like this > >> platform=Linux-3.16.0-50-generic-x86_64-with-redhat-5.11-Final >> sys.platform=linux2 >> os.name=posix There isn't any special os name or platform. A manylinux build is just a Linux build that you intend to satisfy some extra requirements, and there's no way to autodetect your intent :-). So you'll probably want to specify with an envvar or whatever's convenient. > 2) Even when I do change behaviour to try and get a static linkage of say freetype agains my code I'm getting linkage errors. Static linkage is said to be OK in pep 513, but I don't know how to get the multi build code to do it. > > I have checked and there are .a files being produced, but it seems they have the wrong compile time flags my error looks like > >> /opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-CentOS-linux/4.8.2/ld: /usr/local/lib/libfreetype.a(ftinit.o): relocation R_X86_64_32 against `tt_driver_class' can not be used when making a shared object; recompile with -fPIC > > > I suppose I need to create a library of the .o files created for the shared build. Then I can link against that .a rather than the non shared one. Yep. Many build systems assume that static libraries will only be linked into executables, not into shared libraries, so they have the wrong defaults for what you're trying to do. It might be enough just to force -fPIC into the compiler command line, by setting CFLAGS or whatever this build system respects. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: