[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Nathaniel Smith]

On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower at python.org> wrote:
>
> On 10Apr2019 1917, Nathaniel Smith wrote:
> > It sounds like --with-pydebug has accumulated a big grab bag of
> > unrelated features, mostly stuff that was useful at some point for
> > some CPython dev trying to debug CPython itself? It's clearly not
> > designed with end users as the primary audience, given that no-one
> > knows what it actually does and that it makes third-party extensions
> > really awkward to run. If that's right then I think Victor's plan of
> > to sort through what it's actually doing makes a lot of sense,
> > especially if we can remove the ABI breaking stuff, since that causes
> > a disproportionate amount of trouble.
>
> Does it really cause a "disproportionate" amount of trouble? It's
> definitely not meant for anyone who isn't working on C code, whether in
> CPython, an extension or a host application. If you want to use
> third-party extensions and are not able to rebuild them, that's a very
> good sign that you probably shouldn't be on the debug build at all.

Well, here's what I mean by "disproportionate". Some of the costs of
the ABI divergence are:

- The first time I had to debug a C extension, I wasted a bunch of
time trying to figure out how I was supposed to use Debian's
'python-dbg' package (the --with-pydebug build), before eventually
figuring out that it was a red herring and what I actually wanted was
the -dbgsym package (their equivalent of MSVC's /Zi /DEBUG files).

- The extension loading machinery has extra code and complexity to
track the two different ABIs. The package ecosystem does too, e.g.
distutils needs to name extensions appropriately, and we need special
wheel tags, and pip needs code to handle these tags:
    https://github.com/pypa/pip/blob/54b6a91405adc79cdb8a2954e9614d6860799ccb/src/pip/_internal/pep425tags.py#L106-L109

- If you want some of the features of --with-pydebug that don't change
the ABI, then you still have to rebuild third-party extensions to get
at them, and that's a significant hassle. (I could do it if I had to,
but my time has value.)

- Everyone who uses ctypes to access a PyObject* has to include some
extra hacks to handle the difference between the regular and debug
ABIs. There are a few different versions that get copy/pasted around
as folklore, and they're all pretty obscure. For example:
    https://github.com/pallets/jinja/blob/fd89fed7456e755e33ba70674c41be5ab222e193/jinja2/debug.py#L317-L334
    https://github.com/johndpope/sims4-ai-engine/blob/865212e841c716dc4364e0dba286f02af8d716e8/core/framewrapper.py#L12-L41
    https://github.com/python-trio/trio/blob/862ced04e1f19287e098380ed8a0635004c36dd1/trio/_core/_multierror.py#L282
  And then if you want to test this code, it means you have to add a
--with-pydebug build to your CI infrastructure...

I don't know how many people use Py_TRACE_REFS, but if we can't find
anyone on python-dev who uses it then it must be pretty rare. If
dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
all the stuff above, then that seems like a pretty good trade! But
maybe the Windows C runtime issue will foil this...

> >> The reason we ship debug Python binaries is because debug builds use a
> >> different C Runtime, so if you do a debug build of an extension module
> >> you're working on it won't actually work with a non-debug build of CPython.
> >
> > ...But this is an important point. I'd forgotten that MSVC has a habit
> > of changing the entire C runtime when you turn on the compiler's
> > debugging mode.
>
> Technically they are separate options, but most project files are
> configured such that *their* Debug/Release switch affects both the
> compiler options (optimization) and the linker options (C runtime linkage).

So how do other projects handle this? I guess historically the main
target audience for Visual Studio was folks building monolithic apps,
where you can just rebuild everything with whatever options you want,
and compared to that Python extensions are messier. But Python isn't
the only project in this boat. Do ruby, nodejs, R, etc., all provide
separate debug builds with incompatible ABIs on Windows, and propagate
that information throughout their module/package ecosystem?

-n

--
Nathaniel J. Smith -- https://vorpus.org