[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Steve Dower]

On 12Apr.2019 1643, Nathaniel Smith wrote:
> On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower at python.org> wrote:
>>
>> On 10Apr2019 1917, Nathaniel Smith wrote:
> I don't know how many people use Py_TRACE_REFS, but if we can't find
> anyone on python-dev who uses it then it must be pretty rare. If
> dropping Py_TRACE_REFS would let us converge the ABIs and get rid of
> all the stuff above, then that seems like a pretty good trade! But
> maybe the Windows C runtime issue will foil this...

The very first question I asked was whether this would let us converge
the ABIs, and the answer was "no".

Otherwise I'd have said go for it, despite the C runtime issues.

>>>> The reason we ship debug Python binaries is because debug builds use a
>>>> different C Runtime, so if you do a debug build of an extension module
>>>> you're working on it won't actually work with a non-debug build of CPython.
>>>
>>> ...But this is an important point. I'd forgotten that MSVC has a habit
>>> of changing the entire C runtime when you turn on the compiler's
>>> debugging mode.
>>
>> Technically they are separate options, but most project files are
>> configured such that *their* Debug/Release switch affects both the
>> compiler options (optimization) and the linker options (C runtime linkage).
> 
> So how do other projects handle this? I guess historically the main
> target audience for Visual Studio was folks building monolithic apps,
> where you can just rebuild everything with whatever options you want,
> and compared to that Python extensions are messier. But Python isn't
> the only project in this boat. Do ruby, nodejs, R, etc., all provide
> separate debug builds with incompatible ABIs on Windows, and propagate
> that information throughout their module/package ecosystem?

Mostly I hear complaints about those languages *not* providing any help
here. Python is renowned for having significantly better Windows support
than any of them, so they're the wrong comparison to make in my opinion.
Arguing that we should regress because other languages haven't caught up
to us yet makes no sense.

The tools that are better than Python typically don't ship debug builds
either, unless you specifically request them. But they also don't leak
their implementation details all over the place. If we had a better C
API, we wouldn't have users who needed to match ABIs.

For the most part, disabling optimizations in your own extension but
using the non-debug ABI is sufficient, and if you're having to deal with
other people's packages then maybe you don't have any choice (though I
do know of people who have built debug versions of numpy before - turns
out Windows developers are often just as capable as non-Windows
developers when it comes to building things ;) ). And yes, they could
also build CPython from source as well to get the debug ABI, or get the
debug symbols, but I saw enough need that I decided it was worth the
effort to just solve that problem. 250k downloads a month is enough to
justify it for me.

Not to bring the packaging discussions to another venue, but maybe this
is yet another area we need to stop pretending that we're able to solve
every single problem with just the tools we already have available?
People who want debug builds of packages can build them themselves, even
numpy and scipy, they don't need us to preemptively do all their work
for them. But we can (and should) help short-cut unnecessary effort or
research by providing helpful tools and instruction.

Cheers,
Steve