[Wheel-builders] Help with building/using MySQL-python-embedded

[Vitaly Kruglikov]

On 9/15/16, 9:42 PM, "Wheel-builders on behalf of Amit Saha"
<wheel-builders-bounces+vkruglikov=numenta.com at python.org on behalf of
amitsaha.in at gmail.com> wrote:

>Hi all,
>
>I am trying to build manylinux1 wheels for
>https://pypi.python.org/pypi/MySQL-python-embedded/1.2.5
>
>Here is my script which I am running inside the build container:
>
>#!/bin/bash
># Script modified from https://github.com/pypa/python-manylinux-demo
>set -e -x
>
>yum install -y make zlib-devel openssl-devel libaio libaio-devel
>
>wget 
>http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.51.tar.gz/from/http
>://mysql.he.net/
>tar -zxvf mysql-5.1.51.tar.gz
>cd /mysql-5.1.51
>CFLAGS=-fPIC CXXFLAGS=-fPIC ./configure
>make install
>cd libmysqld
>make install
>cd /
>
># Compile wheels
>for PYBIN in /opt/python/cp27*/bin; do
>    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/mysql ${PYBIN}/pip
>install MySQL-Python==1.2.5 --no-index -f /mysql-python-wheels
>    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/mysql ${PYBIN}/pip
>wheel /workspace/ -w wheelhouse/
>done
>
># Bundle external shared libraries into the wheels
>#ls wheelhouse/*
>for whl in wheelhouse/*linux*.whl; do
>    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/mysql auditwheel
>repair $whl -w /workspace/wheelhouse/
>done
>
># Install packages and test
>for PYBIN in /opt/python/cp27*/bin/; do
>    ${PYBIN}/pip install --no-index mysql-python-embedded -f
>/workspace/wheelhouse
>    ${PYBIN}/python -c "import MySQLdb_embedded;
>MySQLdb_embedded.start_server()" || true
>done
>
>
>
>The wheels build fine (example):
>
>Building wheels for collected packages: MySQL-python-embedded,
>MySQL-python
>  Running setup.py bdist_wheel for MySQL-python-embedded ... done
>  Stored in directory: /wheelhouse
>  Running setup.py bdist_wheel for MySQL-python ... done
>  Stored in directory: /wheelhouse
>Successfully built MySQL-python-embedded MySQL-python
>+ for whl in 'wheelhouse/*linux*.whl'
>+ 
>LD_LIBRARY_PATH=/opt/rh/devtoolset-2/root/usr/lib64:/opt/rh/devtoolset-2/r
>oot/usr/lib:/usr/local/lib64:/usr/local/lib:/usr/local/lib/mysql
>+ auditwheel repair
>wheelhouse/MySQL_python-1.2.5-cp27-cp27m-linux_x86_64.whl -w
>/workspace/wheelhouse/
>Repairing MySQL_python-1.2.5-cp27-cp27m-linux_x86_64.whl
>Grafting: /lib64/libz.so.1.2.3 -> .libs_mysql/libz-a147dcb0.so.1.2.3
>Grafting: /usr/local/lib/mysql/libmysqlclient_r.so.16.0.0 ->
>.libs_mysql/libmysqlclient_r-0bea0d7c.so.16.0.0
>Setting RPATH: _mysql.so to "$ORIGIN/.libs_mysql"
>Previous filename tags: linux_x86_64
>New filename tags: manylinux1_x86_64
>Previous WHEEL info tags: cp27-cp27m-linux_x86_64
>New WHEEL info tags: cp27-cp27m-manylinux1_x86_64
>
>Fixed-up wheel written to
>/workspace/wheelhouse/MySQL_python-1.2.5-cp27-cp27m-manylinux1_x86_64.whl
>
>
>Now when I import it inside the same build container:
>
> /opt/python/cp27-cp27mu/bin//python -c 'import MySQLdb_embedded;
>MySQLdb_embedded.start_server()'
>Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File 
>"/opt/python/cp27-cp27mu/lib/python2.7/site-packages/MySQLdb_embedded/__in
>it__.py",
>line 12, in <module>
>    import _mysql_embedded
>ImportError: 
>/opt/python/cp27-cp27mu/lib/python2.7/site-packages/_mysql_embedded.so:
>undefined symbol: __cxa_pure_virtual
>
>
>The reason I download and compile mysql 5.151 from source is mysql
>python embedded needs to statically link to the libmysqld.a library
>which on CentOS5 can only be done as far as I have found out by hand
>compiling above. This is also the reason I use CFLAGS=-fPIC
>CXXFLAGS=-fPIC when running configure.
>
>Not sure what I am doing wrong or what I should be looking at next.
>Any suggestions will be greatly appreciated.
>
>Thank you.
>
>Best Wishes,
>Amit.
>
>
>-- 
>http://echorand.me
>_______________________________________________
>Wheel-builders mailing list
>Wheel-builders at python.org
>https://mail.python.org/mailman/listinfo/wheel-builders


The explanation in 
http://stackoverflow.com/questions/920500/what-is-the-purpose-of-cxa-pure-v
irtual would suggest that you might not be linking with some necessary
default libraries that came with your development environment. While you
can build a shared library on a posix system that doesn¹t have all the
referenced symbols defined, when such a library is loaded into an
application, the runtime loader/linker will signal a failure if not all
symbols were able to be resolved at this time. It seems like in your case,
the default implementation of `__cxa_pure_virtual` was not linked into
your extension .so.

Not related to your problem, but critically important concepts to be aware
of when building python extensions in general and manylinux wheels
specifically are symbol visibility, symbol preemption, and ABI
compatibility (the latter if you¹re compiling from C++ code). If you don¹t
control symbol visibility, you may cause unpredictable behavior either by
having your symbols preempted by a previously-loaded extension or
preempting matching symbols in another subsequently-loaded extension. In a
python extension .so, all symbols must be hidden, except the extension¹s
init function. This may be accomplished with a combination of compile-time
visibility flags (e.g.,
https://github.com/numenta/nupic.core/blob/a4a5d3fd62a11a7d9b9094122d49a9be
6889efd0/CommonCompilerConfig.cmake#L258-L259) and an export map (e.g.,
https://github.com/numenta/nupic.core/blob/0777644933fb8dda5c7c015730cf3717
bea8f724/src/CMakeLists.txt#L786-L815). Here is a summary of the measures
that I took to solve the symbol and ABI compatibility issues that I
encountered: 
https://discourse.numenta.org/t/segmentation-fault-while-running-basic-swar
m/877/24?u=vkruglikov. You may also need to link with certain standard
libraries statically - see comments in
https://github.com/numenta/nupic.core/blob/a4a5d3fd62a11a7d9b9094122d49a9be
6889efd0/CommonCompilerConfig.cmake#L185-L197.

Good luck!

Vitaly