[Wheel-builders] Problem with finding python library via CMake

[Clarkson, Matt]

Hi there,

Thanks JC. Makes sense.

M


From: Jean-Christophe Fillion-Robin <jcfr at kitware.com>
Date: Wednesday, 23 January 2019 at 14:56
To: "Clarkson, Matt" <m.clarkson at ucl.ac.uk>
Cc: "wheel-builders at python.org" <wheel-builders at python.org>
Subject: Re: [Wheel-builders] Problem with finding python library via CMake

Also to ensure nothing link against Python, you could set PYTHON_LIBRARY CMake cache variable to an empty file with a meaningful name (for example /path/to libpython-not-needed-symbols-exported-by-interpreter), this would allow to catch where weak linking is missing.

For an example, see here<https://github.com/InsightSoftwareConsortium/ITKPythonPackage/blob/9b799f01297e475f44e100fd2afd6b13b5ccb869/scripts/macpython-build-common.sh#L33-L47>

On Wed, Jan 23, 2019 at 9:52 AM Jean-Christophe Fillion-Robin <jcfr at kitware.com<mailto:jcfr at kitware.com>> wrote:
Hi Matt,

(1) On linux, you should not link against the library. For that reason, it is not available in manylinux.

To provide some background:

-----------------
Since the python interpreter exports its symbol (see [1]), python modules should not link against any python libraries.
To ensure it is not the case, we configure the project using an empty file as python library.

 [1] "Note that libpythonX.Y.so.1 is not on the list of libraries that a manylinux1 extension is allowed to link to. Explicitly linking to libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF linking works, extension modules that are loaded into the interpreter automatically get access to all of the interpreter's symbols, regardless of whether or not the extension itself is explicitly linked against libpython. [...]"

Source: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1
-----------------



(2) Updated CMake

While this would not solve this particular problem, there are python wheels available for CMake.
See https://pypi.org/pypi/cmake/

And also for ninja build tool (quite usual for automatic parallelization of the build). Note that the version distributed on pypi is patched<https://github.com/kitware/ninja#readme> version that include the jobserver capability as well as as fortran support .
See https://pypi.org/project/ninja/


(3) Streamlined build

I suggest you look into scikit-build, this provide a nice integration of CMake and python setuptools.
See http://scikit-build.readthedocs.io/


(4) CI and upload to PyPI

For an example of project, consider looking at
https://github.com/amueller/word_cloud


Hth
Jc






On Wed, Jan 23, 2019 at 9:41 AM Clarkson, Matt <m.clarkson at ucl.ac.uk<mailto:m.clarkson at ucl.ac.uk>> wrote:
Hi there,

If someone could help me, I would be eternally grateful.

I'm currently creating:
https://github.com/MattClarkson/CMakeCatchTemplate

as a template project, from which others can generate their own C++ CMake, Catch, CTest project, and I'm providing support for building Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via manylinux.

I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks like it's working:
https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507

However, if you look at a Linux build, for example:
https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399

and line 599, we see that CMake is finding the right python executable, and the right python include dir, but picking up /usr/lib64/libpython2.4 instead of the required version 3.5.5. This means, when I pip install on another independent machine, downloading the wheel from pypi, I get a library not found error. I should say at this point, that I'm actually using the opencv-python docker image provided here: quay.io/skvark/manylinux<http://quay.io/skvark/manylinux>, but this is essentially the same as manylinux, with these additions: https://github.com/skvark/opencv-python/tree/master/docker, but from a python perspective, its the same.

So, my question (as I currently know almost nothing about docker):

a) Does the docker image have all python libraries installed? If so, where are they? Is it worth trying to coax cmake 3.9 to find the right one? At the moment I do not yet know (and have limited time today), if I can run or log into the docker container to find the libraries.

b) Or do we just create our own docker image, with a more up to date version of cmake?

If you have any advice, I'd be grateful, as I'm new to wheel building.

Thanks

Matt


_______________________________________________
Wheel-builders mailing list
Wheel-builders at python.org<mailto:Wheel-builders at python.org>
https://mail.python.org/mailman/listinfo/wheel-builders
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20190123/8010d854/attachment-0001.html>