[Wheel-builders] Problem with finding python library via CMake

[Jean-Christophe Fillion-Robin]

Also to ensure nothing link against Python, you could set PYTHON_LIBRARY
CMake cache variable to an empty file with a meaningful name (for example
/path/to libpython-not-needed-symbols-exported-by-interpreter), this would
allow to catch where weak linking is missing.

For an example, see here
<https://github.com/InsightSoftwareConsortium/ITKPythonPackage/blob/9b799f01297e475f44e100fd2afd6b13b5ccb869/scripts/macpython-build-common.sh#L33-L47>

On Wed, Jan 23, 2019 at 9:52 AM Jean-Christophe Fillion-Robin <
jcfr at kitware.com> wrote:

> Hi Matt,
>
> (1) On linux, you should not link against the library. For that reason, it
> is not available in manylinux.
>
> To provide some background:
>
> -----------------
> Since the python interpreter exports its symbol (see [1]), python modules
> should not link against any python libraries.
> To ensure it is not the case, we configure the project using an empty file
> as python library.
>
>  [1] "Note that libpythonX.Y.so.1 is not on the list of libraries that a
> manylinux1 extension is allowed to link to. Explicitly linking to
> libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF linking
> works, extension modules that are loaded into the interpreter automatically
> get access to all of the interpreter's symbols, regardless of whether or
> not the extension itself is explicitly linked against libpython. [...]"
>
> Source: https://www.python.org/dev/peps/pep-0513/#libpythonx-y-so-1
> -----------------
>
>
>
> (2) Updated CMake
>
> While this would not solve this particular problem, there are python
> wheels available for CMake.
> See https://pypi.org/pypi/cmake/
>
> And also for ninja build tool (quite usual for automatic parallelization
> of the build). Note that the version distributed on pypi is patched
> <https://github.com/kitware/ninja#readme> version that include the
> jobserver capability as well as as fortran support .
> See https://pypi.org/project/ninja/
>
>
> (3) Streamlined build
>
> I suggest you look into scikit-build, this provide a nice integration of
> CMake and python setuptools.
> See http://scikit-build.readthedocs.io/
>
>
> (4) CI and upload to PyPI
>
> For an example of project, consider looking at
> https://github.com/amueller/word_cloud
>
>
> Hth
> Jc
>
>
>
>
>
>
> On Wed, Jan 23, 2019 at 9:41 AM Clarkson, Matt <m.clarkson at ucl.ac.uk>
> wrote:
>
>> Hi there,
>>
>> If someone could help me, I would be eternally grateful.
>>
>> I'm currently creating:
>> https://github.com/MattClarkson/CMakeCatchTemplate
>>
>> as a template project, from which others can generate their own C++
>> CMake, Catch, CTest project, and I'm providing support for building
>> Boost.Python, PyBind11, and hence wheels, via Appveyor, and Travis, via
>> manylinux.
>>
>> I'm currently on Issue #45, and branch: 45-pypi, and the Travis log looks
>> like it's working:
>> https://travis-ci.com/MattClarkson/CMakeCatchTemplate/builds/98215507
>>
>> However, if you look at a Linux build, for example:
>> https://travis-ci.com/MattClarkson/CMakeCatchTemplate/jobs/172197399
>>
>> and line 599, we see that CMake is finding the right python executable,
>> and the right python include dir, but picking up /usr/lib64/libpython2.4
>> instead of the required version 3.5.5. This means, when I pip install on
>> another independent machine, downloading the wheel from pypi, I get a
>> library not found error. I should say at this point, that I'm actually
>> using the opencv-python docker image provided here:
>> quay.io/skvark/manylinux, but this is essentially the same as manylinux,
>> with these additions:
>> https://github.com/skvark/opencv-python/tree/master/docker, but from a
>> python perspective, its the same.
>>
>> So, my question (as I currently know almost nothing about docker):
>>
>> a) Does the docker image have all python libraries installed? If so,
>> where are they? Is it worth trying to coax cmake 3.9 to find the right one?
>> At the moment I do not yet know (and have limited time today), if I can run
>> or log into the docker container to find the libraries.
>>
>> b) Or do we just create our own docker image, with a more up to date
>> version of cmake?
>>
>> If you have any advice, I'd be grateful, as I'm new to wheel building.
>>
>> Thanks
>>
>> Matt
>>
>>
>> _______________________________________________
>> Wheel-builders mailing list
>> Wheel-builders at python.org
>> https://mail.python.org/mailman/listinfo/wheel-builders
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20190123/98def7cc/attachment.html>