[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Matthias Klose]

On 25.04.19 08:31, Nathaniel Smith wrote:
> You don't necessarily need rpath actually. The Linux loader has a
> bug/feature where once it has successfully loaded a library with a given
> soname, then any future requests for that soname within the same process
> will automatically return that same library, regardless of rpath settings
> etc. So as long as the main interpreter has loaded libpython.whatever from
> the correct directory, then extension modules will all get that same
> version. The rpath won't matter at all.
> 
> It is annoying in general that on Linux, we have these two different ways
> to build extension modules. It definitely violates TOOWTDI :-). It would be
> nice at some point to get rid of one of them.
> 
> Note that we can't get rid of the two different ways entirely though – on
> Windows, extension modules *must* link to libpython.dll, and on macOS,
> extension modules *can't* link to libpython.dylib. So the best we can hope
> for is to make Linux consistently do one of these, instead of supporting
> both.
> 
> In principle, having extension modules link to libpython.so is a good
> thing. Suppose that someone wants to dynamically load the python
> interpreter into their program as some kind of plugin. (Examples: Apache's
> mod_python, LibreOffice's support for writing macros in Python.) It would
> be nice to be able to load python2 and python3 simultaneously into the same
> process as distinct plugins. And this is totally doable in theory, *but* it
> means that you can't assume that the interpreter's symbols will be
> automagically injected into extension modules, so it's only possible if
> extension modules link to libpython.so.
> 
> In practice, extension modules have never consistently linked to
> libpython.so, so everybody who loads the interpreter as a plugin has
> already worked around this. Specifically, they use RTLD_GLOBAL to dump all
> the interpreter's symbols into the global namespace. This is why you can't
> have python2 and python3 mod_python at the same time in the same Apache.
> And since everyone is already working around this, linking to libpython.so
> currently has zero benefit... in fact manylinux wheels are actually
> forbidden to link to libpython.so, because this is the only way to get
> wheels that work on every interpreter.

extensions in Debian/Ubuntu packages are not linked against libpython.so, but
the main reason here is that sometimes you have to extensions built in
transition periods like for 3.6 and 3.7. And this is also the default when not
configuring with --enable-shared.