[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Victor Stinner]

Le jeu. 25 avr. 2019 à 09:34, Matthias Klose <doko at ubuntu.com> a écrit :
> there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;)  So
> depending on the package maintainer, you already have that available, but it is
> extra maintenance cost.  Simplifying that would be a good idea.

Fedora provides "debuginfo" for all binarry packages (like numpy), but
that's different from a debug build. Usually, C code of packages are
optimized by gcc -O2 or even gcc -O3 which makes the debugging
experience very painful: gdb fails to read C local variables and just
say "<optimized out>". To debug internals, you want a debug build
compiled by gcc -Og or (better IMHO) gcc -O0.

If you want to inspect *Python* internals but you don't need to
inspect numpy internals, being able to run a release numpy on a debug
Python is convenient.

With an additional change on SOABI (I will open a separated issue for
that), my PR 12946 (no longer link C extensions to libpython) allows
to load lxml built in release mode in a Python built in debug mode!
That's *very* useful for debugging. I show an example of the gdb
experience with a release Python vs debug Python:

https://bugs.python.org/issue21536#msg340821

With a release Python, the basic function "py-bt" works as expected,
but inspecting Python internals doesn't work: most local C variables
are "optimized out" :-(

With a debug Python, the debugging experience is *much* better: it's
possible to inspect Python internals!


> However I still
> would like to be able to have "debug" and "non-debug" builds co-installable at
> the same time.

One option is to keep "d" flag in the SOABI so C extensions get a
different SO filename (no change compared to Python 3.7):
"NAME.cpython-38-x86_64-linux-gnu.so" for release vs
"NAME.cpython-38d-x86_64-linux-gnu.so" for debug, debug gets "d"
suffix ("cpython-38" vs "cpython-38d").

*But* modify importlib when Python is compiled in debug mode to look
also to SO without the "d" suffix: first try load
"NAME.cpython-38d-x86_64-linux-gnu.so" (debug: "d" suffix). If there
is no match, look for "NAME.cpython-38-x86_64-linux-gnu.so" (release:
no suffix). Since the ABI is now compatible in Python 3.8, it should
"just work" :-)

>From a Linux packager perspective, nothing changes ;-) We can still
provide "apt install python3-numpy-dbg" (debug) which can is
co-installable with "apt install python3-numpy" (release).

The benefit is that it will be possible to load C extensions which are
only available in the release flavor with a debug Python ;-)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.