From ivaylo_angelov at gmx.de Sun Jan 20 16:22:34 2019 From: ivaylo_angelov at gmx.de (Ivaylo Angelov) Date: Sun, 20 Jan 2019 22:22:34 +0100 Subject: [Wheel-builders] use of machomachomangler Message-ID: An HTML attachment was scrubbed... URL: From njs at pobox.com Mon Jan 21 05:07:44 2019 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 21 Jan 2019 02:07:44 -0800 Subject: [Wheel-builders] use of machomachomangler In-Reply-To: References: Message-ID: On Mon, Jan 21, 2019 at 2:03 AM Ivaylo Angelov wrote: > > Dear Mr. Smith, > > writing a program, I am right now facing a similar problem as the one > you have described on stackoverflow regarding the modification of PE import tables. > I took a look into the github repository you have mentioned there and would like > to use your program, but I want to make sure, that I do it properly, because the scenario in > my case is slightly different: > > I have a dll named containing the in the import table. > All I want to do, is to replace by ... Assuming you have a python 3 install, with the machomachomangler package installed, It should be something like: python3 -m machomachomangler.cmd.redll A.dll A-patched.dll B.dll C.dll This leaves your original A.dll unchanged, but writes out a new file called "A-patched.dll". So you probably want to follow that by renaming A.dll ? A-original.dll, and A-patched.dll ? A.dll. Unfortunately I haven't had time to look at this for a few years, so if you run into more trouble I may not be able to help, but I think that should work? Good luck! -n -- Nathaniel J. Smith -- https://vorpus.org From m.clarkson at ucl.ac.uk Wed Jan 23 03:33:00 2019 From: m.clarkson at ucl.ac.uk (Clarkson, Matt) Date: Wed, 23 Jan 2019 08:33:00 +0000 Subject: [Wheel-builders] Problem with finding python library via CMake Message-ID: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> Hi there, If someone could help me, I would be eternally grateful. I'm currently creating: https://github.com/MattClarkson/CMakeCatchTemplate as a template project, from which others can generate their own C++ CMake, Catch, CTest project, and I'm providing support for building Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux. I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks like it's working: https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 However, if you look at a Linux build, for example: https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 and line 599, we see that CMake is finding the right python executable, and the right python include dir, but picking up /usr/lib64/libpython2.4 instead of the required version 3.5.5. This means, when I pip install on another independent machine, downloading the wheel from pypi, I get a library not found error. I should say at this point, that I'm actually using the opencv-python docker image provided here: quay.io/skvark/manylinux, but this is essentially the same as manylinux, with these additions: https://github.com/skvark/opencv-python/tree/master/docker, but from a python perspective, its the same. So, my question (as I currently know almost nothing about docker): a) Does the docker image have all python libraries installed? If so, where are they? Is it worth trying to coax cmake 3.9 to find the right one? At the moment I do not yet know (and have limited time today), if I can run or log into the docker container to find the libraries. b) Or do we just create our own docker image, with a more up to date version of cmake? If you have any advice, I'd be grateful, as I'm new to wheel building. Thanks Matt From matthew.brett at gmail.com Wed Jan 23 09:52:54 2019 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 23 Jan 2019 14:52:54 +0000 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> Message-ID: Hi, On Wed, Jan 23, 2019 at 2:41 PM Clarkson, Matt wrote: > > Hi there, > > If someone could help me, I would be eternally grateful. > > I'm currently creating: > https://github.com/MattClarkson/CMakeCatchTemplate > > as a template project, from which others can generate their own C++ CMake, Catch, CTest project, and I'm providing support for building Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux. > > I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks like it's working: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 > > However, if you look at a Linux build, for example: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 > > and line 599, we see that CMake is finding the right python executable, and the right python include dir, but picking up /usr/lib64/libpython2.4 instead of the required version 3.5.5. This means, when I pip install on another independent machine, downloading the wheel from pypi, I get a library not found error. I should say at this point, that I'm actually using the opencv-python docker image provided here: quay.io/skvark/manylinux, but this is essentially the same as manylinux, with these additions: https://github.com/skvark/opencv-python/tree/master/docker, but from a python perspective, its the same. > > So, my question (as I currently know almost nothing about docker): > > a) Does the docker image have all python libraries installed? If so, where are they? Is it worth trying to coax cmake 3.9 to find the right one? At the moment I do not yet know (and have limited time today), if I can run or log into the docker container to find the libraries. > > b) Or do we just create our own docker image, with a more up to date version of cmake? > > If you have any advice, I'd be grateful, as I'm new to wheel building. I think CMake is particularly prone to requiring the libpython library, which is deliberately not part of the Manylinux spec. See the discussion here: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 Cheers, Matthew From jcfr at kitware.com Wed Jan 23 09:52:28 2019 From: jcfr at kitware.com (Jean-Christophe Fillion-Robin) Date: Wed, 23 Jan 2019 09:52:28 -0500 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> Message-ID: Hi Matt, (1) On linux, you should not link against the library. For that reason, it is not available in manylinux. To provide some background: ----------------- Since the python interpreter exports its symbol (see [1]), python modules should not link against any python libraries. To ensure it is not the case, we configure the project using an empty file as python library. [1] "Note that libpythonX.Y.so.1 is not on the list of libraries that a manylinux1 extension is allowed to link to. Explicitly linking to libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF linking works, extension modules that are loaded into the interpreter automatically get access to all of the interpreter's symbols, regardless of whether or not the extension itself is explicitly linked against libpython. [...]" Source: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 ----------------- (2) Updated CMake While this would not solve this particular problem, there are python wheels available for CMake. See https://pypi.org/pypi/cmake/ And also for ninja build tool (quite usual for automatic parallelization of the build). Note that the version distributed on pypi is patched version that include the jobserver capability as well as as fortran support . See https://pypi.org/project/ninja/ (3) Streamlined build I suggest you look into scikit-build, this provide a nice integration of CMake and python setuptools. See http://scikit-build.readthedocs.io/ (4) CI and upload to PyPI For an example of project, consider looking at https://github.com/amueller/word_cloud Hth Jc On Wed, Jan 23, 2019 at 9:41 AM Clarkson, Matt wrote: > Hi there, > > If someone could help me, I would be eternally grateful. > > I'm currently creating: > https://github.com/MattClarkson/CMakeCatchTemplate > > as a template project, from which others can generate their own C++ CMake, > Catch, CTest project, and I'm providing support for building Boost.Python, > PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux. > > I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks > like it's working: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 > > However, if you look at a Linux build, for example: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 > > and line 599, we see that CMake is finding the right python executable, > and the right python include dir, but picking up /usr/lib64/libpython2.4 > instead of the required version 3.5.5. This means, when I pip install on > another independent machine, downloading the wheel from pypi, I get a > library not found error. I should say at this point, that I'm actually > using the opencv-python docker image provided here: > quay.io/skvark/manylinux, but this is essentially the same as manylinux, > with these additions: > https://github.com/skvark/opencv-python/tree/master/docker, but from a > python perspective, its the same. > > So, my question (as I currently know almost nothing about docker): > > a) Does the docker image have all python libraries installed? If so, where > are they? Is it worth trying to coax cmake 3.9 to find the right one? At > the moment I do not yet know (and have limited time today), if I can run or > log into the docker container to find the libraries. > > b) Or do we just create our own docker image, with a more up to date > version of cmake? > > If you have any advice, I'd be grateful, as I'm new to wheel building. > > Thanks > > Matt > > > _______________________________________________ > Wheel-builders mailing list > Wheel-builders at python.org > https://mail.python.org/mailman/listinfo/wheel-builders > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcfr at kitware.com Wed Jan 23 09:55:55 2019 From: jcfr at kitware.com (Jean-Christophe Fillion-Robin) Date: Wed, 23 Jan 2019 09:55:55 -0500 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> Message-ID: Also to ensure nothing link against Python, you could set PYTHON_LIBRARY CMake cache variable to an empty file with a meaningful name (for example /path/to libpython-not-needed-symbols-exported-by-interpreter), this would allow to catch where weak linking is missing. For an example, see here On Wed, Jan 23, 2019 at 9:52 AM Jean-Christophe Fillion-Robin < jcfr at kitware.com> wrote: > Hi Matt, > > (1) On linux, you should not link against the library. For that reason, it > is not available in manylinux. > > To provide some background: > > ----------------- > Since the python interpreter exports its symbol (see [1]), python modules > should not link against any python libraries. > To ensure it is not the case, we configure the project using an empty file > as python library. > > [1] "Note that libpythonX.Y.so.1 is not on the list of libraries that a > manylinux1 extension is allowed to link to. Explicitly linking to > libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF linking > works, extension modules that are loaded into the interpreter automatically > get access to all of the interpreter's symbols, regardless of whether or > not the extension itself is explicitly linked against libpython. [...]" > > Source: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 > ----------------- > > > > (2) Updated CMake > > While this would not solve this particular problem, there are python > wheels available for CMake. > See https://pypi.org/pypi/cmake/ > > And also for ninja build tool (quite usual for automatic parallelization > of the build). Note that the version distributed on pypi is patched > version that include the > jobserver capability as well as as fortran support . > See https://pypi.org/project/ninja/ > > > (3) Streamlined build > > I suggest you look into scikit-build, this provide a nice integration of > CMake and python setuptools. > See http://scikit-build.readthedocs.io/ > > > (4) CI and upload to PyPI > > For an example of project, consider looking at > https://github.com/amueller/word_cloud > > > Hth > Jc > > > > > > > On Wed, Jan 23, 2019 at 9:41 AM Clarkson, Matt > wrote: > >> Hi there, >> >> If someone could help me, I would be eternally grateful. >> >> I'm currently creating: >> https://github.com/MattClarkson/CMakeCatchTemplate >> >> as a template project, from which others can generate their own C++ >> CMake, Catch, CTest project, and I'm providing support for building >> Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via >> manylinux. >> >> I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks >> like it's working: >> https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 >> >> However, if you look at a Linux build, for example: >> https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 >> >> and line 599, we see that CMake is finding the right python executable, >> and the right python include dir, but picking up /usr/lib64/libpython2.4 >> instead of the required version 3.5.5. This means, when I pip install on >> another independent machine, downloading the wheel from pypi, I get a >> library not found error. I should say at this point, that I'm actually >> using the opencv-python docker image provided here: >> quay.io/skvark/manylinux, but this is essentially the same as manylinux, >> with these additions: >> https://github.com/skvark/opencv-python/tree/master/docker, but from a >> python perspective, its the same. >> >> So, my question (as I currently know almost nothing about docker): >> >> a) Does the docker image have all python libraries installed? If so, >> where are they? Is it worth trying to coax cmake 3.9 to find the right one? >> At the moment I do not yet know (and have limited time today), if I can run >> or log into the docker container to find the libraries. >> >> b) Or do we just create our own docker image, with a more up to date >> version of cmake? >> >> If you have any advice, I'd be grateful, as I'm new to wheel building. >> >> Thanks >> >> Matt >> >> >> _______________________________________________ >> Wheel-builders mailing list >> Wheel-builders at python.org >> https://mail.python.org/mailman/listinfo/wheel-builders >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.clarkson at ucl.ac.uk Wed Jan 23 09:56:11 2019 From: m.clarkson at ucl.ac.uk (Clarkson, Matt) Date: Wed, 23 Jan 2019 14:56:11 +0000 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> Message-ID: Thanks Matthew, Understood. Good job I asked before I started hacking around too much! M ?-----Original Message----- From: Matthew Brett Date: Wednesday, 23 January 2019 at 14:50 To: "Clarkson, Matt" Cc: "wheel-builders at python.org" Subject: Re: [Wheel-builders] Problem with finding python library via CMake Hi, On Wed, Jan 23, 2019 at 2:41 PM Clarkson, Matt wrote: > > Hi there, > > If someone could help me, I would be eternally grateful. > > I'm currently creating: > https://github.com/MattClarkson/CMakeCatchTemplate > > as a template project, from which others can generate their own C++ CMake, Catch, CTest project, and I'm providing support for building Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux. > > I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks like it's working: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 > > However, if you look at a Linux build, for example: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 > > and line 599, we see that CMake is finding the right python executable, and the right python include dir, but picking up /usr/lib64/libpython2.4 instead of the required version 3.5.5. This means, when I pip install on another independent machine, downloading the wheel from pypi, I get a library not found error. I should say at this point, that I'm actually using the opencv-python docker image provided here: quay.io/skvark/manylinux, but this is essentially the same as manylinux, with these additions: https://github.com/skvark/opencv-python/tree/master/docker, but from a python perspective, its the same. > > So, my question (as I currently know almost nothing about docker): > > a) Does the docker image have all python libraries installed? If so, where are they? Is it worth trying to coax cmake 3.9 to find the right one? At the moment I do not yet know (and have limited time today), if I can run or log into the docker container to find the libraries. > > b) Or do we just create our own docker image, with a more up to date version of cmake? > > If you have any advice, I'd be grateful, as I'm new to wheel building. I think CMake is particularly prone to requiring the libpython library, which is deliberately not part of the Manylinux spec. See the discussion here: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 Cheers, Matthew From m.clarkson at ucl.ac.uk Wed Jan 23 10:16:19 2019 From: m.clarkson at ucl.ac.uk (Clarkson, Matt) Date: Wed, 23 Jan 2019 15:16:19 +0000 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> Message-ID: <1CC7FAF1-4C94-43CF-9EEB-25B54545B172@ucl.ac.uk> Hi JC, Last question (I hope) ? does the same apply on Windows? Please excuse my ignorance. M From: Jean-Christophe Fillion-Robin Date: Wednesday, 23 January 2019 at 14:52 To: "Clarkson, Matt" Cc: "wheel-builders at python.org" Subject: Re: [Wheel-builders] Problem with finding python library via CMake Hi Matt, (1) On linux, you should not link against the library. For that reason, it is not available in manylinux. To provide some background: ----------------- Since the python interpreter exports its symbol (see [1]), python modules should not link against any python libraries. To ensure it is not the case, we configure the project using an empty file as python library. [1] "Note that libpythonX.Y.so.1 is not on the list of libraries that a manylinux1 extension is allowed to link to. Explicitly linking to libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF linking works, extension modules that are loaded into the interpreter automatically get access to all of the interpreter's symbols, regardless of whether or not the extension itself is explicitly linked against libpython. [...]" Source: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 ----------------- (2) Updated CMake While this would not solve this particular problem, there are python wheels available for CMake. See https://pypi.org/pypi/cmake/ And also for ninja build tool (quite usual for automatic parallelization of the build). Note that the version distributed on pypi is patched version that include the jobserver capability as well as as fortran support . See https://pypi.org/project/ninja/ (3) Streamlined build I suggest you look into scikit-build, this provide a nice integration of CMake and python setuptools. See http://scikit-build.readthedocs.io/ (4) CI and upload to PyPI For an example of project, consider looking at https://github.com/amueller/word_cloud Hth Jc On Wed, Jan 23, 2019 at 9:41 AM Clarkson, Matt > wrote: Hi there, If someone could help me, I would be eternally grateful. I'm currently creating: https://github.com/MattClarkson/CMakeCatchTemplate as a template project, from which others can generate their own C++ CMake, Catch, CTest project, and I'm providing support for building Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux. I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks like it's working: https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 However, if you look at a Linux build, for example: https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 and line 599, we see that CMake is finding the right python executable, and the right python include dir, but picking up /usr/lib64/libpython2.4 instead of the required version 3.5.5. This means, when I pip install on another independent machine, downloading the wheel from pypi, I get a library not found error. I should say at this point, that I'm actually using the opencv-python docker image provided here: quay.io/skvark/manylinux, but this is essentially the same as manylinux, with these additions: https://github.com/skvark/opencv-python/tree/master/docker, but from a python perspective, its the same. So, my question (as I currently know almost nothing about docker): a) Does the docker image have all python libraries installed? If so, where are they? Is it worth trying to coax cmake 3.9 to find the right one? At the moment I do not yet know (and have limited time today), if I can run or log into the docker container to find the libraries. b) Or do we just create our own docker image, with a more up to date version of cmake? If you have any advice, I'd be grateful, as I'm new to wheel building. Thanks Matt _______________________________________________ Wheel-builders mailing list Wheel-builders at python.org https://mail.python.org/mailman/listinfo/wheel-builders -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcfr at kitware.com Wed Jan 23 10:24:14 2019 From: jcfr at kitware.com (Jean-Christophe Fillion-Robin) Date: Wed, 23 Jan 2019 10:24:14 -0500 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: <1CC7FAF1-4C94-43CF-9EEB-25B54545B172@ucl.ac.uk> References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> <1CC7FAF1-4C94-43CF-9EEB-25B54545B172@ucl.ac.uk> Message-ID: Hi Matt, You can assume that one windows the python dll will be available. (this is the case when using regular python distributions available python.org, anaconda, etc ... ) But not that in that case, the PYTHON_LIBRARY CMake variable will be set to a path like "C:/PythonXY-x64/libs/pythonXY.lib". This corresponds to the import library associated with the pythonXY.dll (only need during compilation) For example, see here Hth Jc On Wed, Jan 23, 2019 at 10:16 AM Clarkson, Matt wrote: > Hi JC, > > > > Last question (I hope) ? does the same apply on Windows? > > > > Please excuse my ignorance. > > > > M > > > > *From: *Jean-Christophe Fillion-Robin > *Date: *Wednesday, 23 January 2019 at 14:52 > *To: *"Clarkson, Matt" > *Cc: *"wheel-builders at python.org" > *Subject: *Re: [Wheel-builders] Problem with finding python library via > CMake > > > > Hi Matt, > > > > (1) On linux, you should not link against the library. For that reason, it > is not available in manylinux. > > > > To provide some background: > > > > ----------------- > > Since the python interpreter exports its symbol (see [1]), python modules > should not link against any python libraries. > > To ensure it is not the case, we configure the project using an empty file > as python library. > > > > [1] "Note that libpythonX.Y.so.1 is not on the list of libraries that a > manylinux1 extension is allowed to link to. Explicitly linking to > libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF linking > works, extension modules that are loaded into the interpreter automatically > get access to all of the interpreter's symbols, regardless of whether or > not the extension itself is explicitly linked against libpython. [...]" > > > > Source: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 > > ----------------- > > > > > > > > (2) Updated CMake > > > > While this would not solve this particular problem, there are python > wheels available for CMake. > > See https://pypi.org/pypi/cmake/ > > > > And also for ninja build tool (quite usual for automatic parallelization > of the build). Note that the version distributed on pypi is patched > version that include the > jobserver capability as well as as fortran support . > > See https://pypi.org/project/ninja/ > > > > > > (3) Streamlined build > > > > I suggest you look into scikit-build, this provide a nice integration of > CMake and python setuptools. > > See http://scikit-build.readthedocs.io/ > > > > > > (4) CI and upload to PyPI > > > > For an example of project, consider looking at > > https://github.com/amueller/word_cloud > > > > > > Hth > > Jc > > > > > > > > > > > > > > On Wed, Jan 23, 2019 at 9:41 AM Clarkson, Matt > wrote: > > Hi there, > > If someone could help me, I would be eternally grateful. > > I'm currently creating: > https://github.com/MattClarkson/CMakeCatchTemplate > > as a template project, from which others can generate their own C++ CMake, > Catch, CTest project, and I'm providing support for building Boost.Python, > PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux. > > I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks > like it's working: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 > > However, if you look at a Linux build, for example: > https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 > > and line 599, we see that CMake is finding the right python executable, > and the right python include dir, but picking up /usr/lib64/libpython2.4 > instead of the required version 3.5.5. This means, when I pip install on > another independent machine, downloading the wheel from pypi, I get a > library not found error. I should say at this point, that I'm actually > using the opencv-python docker image provided here: > quay.io/skvark/manylinux, but this is essentially the same as manylinux, > with these additions: > https://github.com/skvark/opencv-python/tree/master/docker, but from a > python perspective, its the same. > > So, my question (as I currently know almost nothing about docker): > > a) Does the docker image have all python libraries installed? If so, where > are they? Is it worth trying to coax cmake 3.9 to find the right one? At > the moment I do not yet know (and have limited time today), if I can run or > log into the docker container to find the libraries. > > b) Or do we just create our own docker image, with a more up to date > version of cmake? > > If you have any advice, I'd be grateful, as I'm new to wheel building. > > Thanks > > Matt > > > _______________________________________________ > Wheel-builders mailing list > Wheel-builders at python.org > https://mail.python.org/mailman/listinfo/wheel-builders > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.clarkson at ucl.ac.uk Wed Jan 23 09:59:17 2019 From: m.clarkson at ucl.ac.uk (Clarkson, Matt) Date: Wed, 23 Jan 2019 14:59:17 +0000 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> Message-ID: <678517A7-2691-4693-A0D7-9B0416CDA5DC@ucl.ac.uk> Hi there, Thanks JC. Makes sense. M From: Jean-Christophe Fillion-Robin Date: Wednesday, 23 January 2019 at 14:56 To: "Clarkson, Matt" Cc: "wheel-builders at python.org" Subject: Re: [Wheel-builders] Problem with finding python library via CMake Also to ensure nothing link against Python, you could set PYTHON_LIBRARY CMake cache variable to an empty file with a meaningful name (for example /path/to libpython-not-needed-symbols-exported-by-interpreter), this would allow to catch where weak linking is missing. For an example, see here On Wed, Jan 23, 2019 at 9:52 AM Jean-Christophe Fillion-Robin > wrote: Hi Matt, (1) On linux, you should not link against the library. For that reason, it is not available in manylinux. To provide some background: ----------------- Since the python interpreter exports its symbol (see [1]), python modules should not link against any python libraries. To ensure it is not the case, we configure the project using an empty file as python library. [1] "Note that libpythonX.Y.so.1 is not on the list of libraries that a manylinux1 extension is allowed to link to. Explicitly linking to libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF linking works, extension modules that are loaded into the interpreter automatically get access to all of the interpreter's symbols, regardless of whether or not the extension itself is explicitly linked against libpython. [...]" Source: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1 ----------------- (2) Updated CMake While this would not solve this particular problem, there are python wheels available for CMake. See https://pypi.org/pypi/cmake/ And also for ninja build tool (quite usual for automatic parallelization of the build). Note that the version distributed on pypi is patched version that include the jobserver capability as well as as fortran support . See https://pypi.org/project/ninja/ (3) Streamlined build I suggest you look into scikit-build, this provide a nice integration of CMake and python setuptools. See http://scikit-build.readthedocs.io/ (4) CI and upload to PyPI For an example of project, consider looking at https://github.com/amueller/word_cloud Hth Jc On Wed, Jan 23, 2019 at 9:41 AM Clarkson, Matt > wrote: Hi there, If someone could help me, I would be eternally grateful. I'm currently creating: https://github.com/MattClarkson/CMakeCatchTemplate as a template project, from which others can generate their own C++ CMake, Catch, CTest project, and I'm providing support for building Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux. I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks like it's working: https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507 However, if you look at a Linux build, for example: https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399 and line 599, we see that CMake is finding the right python executable, and the right python include dir, but picking up /usr/lib64/libpython2.4 instead of the required version 3.5.5. This means, when I pip install on another independent machine, downloading the wheel from pypi, I get a library not found error. I should say at this point, that I'm actually using the opencv-python docker image provided here: quay.io/skvark/manylinux, but this is essentially the same as manylinux, with these additions: https://github.com/skvark/opencv-python/tree/master/docker, but from a python perspective, its the same. So, my question (as I currently know almost nothing about docker): a) Does the docker image have all python libraries installed? If so, where are they? Is it worth trying to coax cmake 3.9 to find the right one? At the moment I do not yet know (and have limited time today), if I can run or log into the docker container to find the libraries. b) Or do we just create our own docker image, with a more up to date version of cmake? If you have any advice, I'd be grateful, as I'm new to wheel building. Thanks Matt _______________________________________________ Wheel-builders mailing list Wheel-builders at python.org https://mail.python.org/mailman/listinfo/wheel-builders -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jan 23 11:51:18 2019 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Jan 2019 08:51:18 -0800 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> <1CC7FAF1-4C94-43CF-9EEB-25B54545B172@ucl.ac.uk> Message-ID: On Wed, Jan 23, 2019, 07:24 Jean-Christophe Fillion-Robin via Wheel-builders Hi Matt, > > You can assume that one windows the python dll will be available. (this is > the case when using regular python distributions available python.org, > anaconda, etc ... ) > Yeah, on Windows, linking against the python dll is absolutely mandatory for technical reasons, so you can assume it's always there. And on macOS, linking against the python dylib is essentially impossible for technical reasons, so tools should generally be set up to cope with this. The confusing case is Linux, where both options are supported and used on different systems. In practice this means portable packages have to avoid linking against the python .so, because it's not always available. But, you can't assume tools will be set up to handle this, because some tool developers only tested on systems that have a usable python .so. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.clarkson at ucl.ac.uk Wed Jan 23 16:03:23 2019 From: m.clarkson at ucl.ac.uk (Clarkson, Matt) Date: Wed, 23 Jan 2019 21:03:23 +0000 Subject: [Wheel-builders] Problem with finding python library via CMake In-Reply-To: References: <505C735F-BA5B-4E93-875B-CB8A6C43C919@ucl.ac.uk> <1CC7FAF1-4C94-43CF-9EEB-25B54545B172@ucl.ac.uk> Message-ID: Great. Thanks everyone. M From: Nathaniel Smith Date: Wednesday, 23 January 2019 at 16:51 To: Jean-Christophe Fillion-Robin Cc: "Clarkson, Matt" , "wheel-builders at python.org" Subject: Re: [Wheel-builders] Problem with finding python library via CMake On Wed, Jan 23, 2019, 07:24 Jean-Christophe Fillion-Robin via Wheel-builders wrote: Hi Matt, You can assume that one windows the python dll will be available. (this is the case when using regular python distributions available python.org, anaconda, etc ... ) Yeah, on Windows, linking against the python dll is absolutely mandatory for technical reasons, so you can assume it's always there. And on macOS, linking against the python dylib is essentially impossible for technical reasons, so tools should generally be set up to cope with this. The confusing case is Linux, where both options are supported and used on different systems. In practice this means portable packages have to avoid linking against the python .so, because it's not always available. But, you can't assume tools will be set up to handle this, because some tool developers only tested on systems that have a usable python .so. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ehashman at debian.org Wed Jan 23 19:50:58 2019 From: ehashman at debian.org (Elana Hashman) Date: Wed, 23 Jan 2019 19:50:58 -0500 Subject: [Wheel-builders] auditwheel 2.0 has been released! Message-ID: <20190124005058.GA659@silverpine> Greetings programs, Following in the spirit of the pip 19.0 release that went out yesterday, I have released auditwheel 2.0 after a long 2-month testing period. Release notes can be found on GitHub[*] and in the CHANGELOG.md file, although no code changes have been made since 2.0rc1. Go forth and build manylinux2010 wheels (and tell me if I broke your build ^_^;;) Cheers, - e [*]: https://github.com/pypa/auditwheel/releases/tag/2.0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From mousumi.nina.sahu1 at gmail.com Thu Jan 31 00:39:11 2019 From: mousumi.nina.sahu1 at gmail.com (mousumi sahu) Date: Thu, 31 Jan 2019 11:09:11 +0530 Subject: [Wheel-builders] Pre-compiled binaries of HT-Seq Message-ID: Hello, I am willing to use HTSeq for my study. I would be grateful if you can please suggest me where can I get pre-compiled binaries of the tool. -- Sincerely, Mousumi -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Jan 31 09:07:16 2019 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 31 Jan 2019 14:07:16 +0000 Subject: [Wheel-builders] Pre-compiled binaries of HT-Seq In-Reply-To: References: Message-ID: Hi, On Thu, Jan 31, 2019 at 1:29 PM mousumi sahu wrote: > > Hello, > I am willing to use HTSeq for my study. I would be grateful if you can please suggest me where can I get pre-compiled binaries of the tool. This list is more general questions about how to build wheels, policies and so on. If you want wheels for HTSeq, you should contact the author. I see they already provide macOS and Manylinux wheels for some Python versions: https://pypi.org/project/HTSeq/#files Cheers, Matthew