[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

[Steve Dower]

On 12Apr2019 1819, Nathaniel Smith wrote:
> On Fri, Apr 12, 2019 at 5:05 PM Steve Dower <steve.dower at python.org> wrote:
>>
>> On 12Apr.2019 1643, Nathaniel Smith wrote:
>>> On Thu, Apr 11, 2019 at 8:26 AM Steve Dower <steve.dower at python.org> wrote:
>> The very first question I asked was whether this would let us converge
>> the ABIs, and the answer was "no".
>>
>> Otherwise I'd have said go for it, despite the C runtime issues.
> 
> I don't see that in the thread... just Victor saying he isn't sure
> whether there might be other ABI incompatibilities lurking that he
> hasn't found yet. Did I miss something?

"I don't know" means we can't say the APIs are converged, which is a no.

I don't think you missed anything, but just read it through a different 
filter.

> I'm mostly interested in this because of the possibility of converging
> the ABIs. If you think that the C runtime thing isn't a blocker for
> that, then that's useful information. Though obviously we still need
> to figure out whether there are any other blockers :-).
> [SNIP]
> Do you happen to have a list of places where the C API leaks details
> of the underlying CRT?
> 
> (I'm mostly curious because whenever I've looked my conclusion was
> essentially: "Well....... I don't see any places that are *definitely*
> broken, so maybe mixing CRTs is fine? but I have zero confidence that
> I caught everything, so probably better to play it safe?". At least on
> py3 – I know the py2 C API was definitely broken if you mixed CRTs,
> because of the exposed FILE*.)

Not since the discussions about migrating to VS 2015, but a few off the 
top of my head:
* locale
* file descriptors
* stream buffers
* thread locals
* exception [handler] state (yes, there are exceptions used within the 
CRT, and they occasionally intentionally leak out past the C code)
* atexit handlers
* internal callbacks (mostly debug handlers, but since we're talking 
about debugging...)

I'm pretty sure if I did some digging I'd be able to figure out which of 
these come from vcruntime140.dll vs ucrtbase.dll, and then come up with 
some far-too-clever linker options to make some of these more 
consistent, but there's no complete solution other than making sure 
you've got a complete debug or complete release build.

>> For the most part, disabling optimizations in your own extension but
>> using the non-debug ABI is sufficient, and if you're having to deal with
>> other people's packages then maybe you don't have any choice (though I
>> do know of people who have built debug versions of numpy before - turns
>> out Windows developers are often just as capable as non-Windows
>> developers when it comes to building things ;)
> 
> I'm not sure why you think I was implying otherwise? I'm sorry if you
> thought I was attacking your users or something. I did say that I
> thought most users downloading the debug builds were probably confused
> about what they were actually getting, but I didn't mean because they
> were stupid Windows users, I meant because the debug builds are so
> confusing that even folks on the Python core team are confused about
> what they're actually getting.

"Our users", please :)

In my experience, Windows developers just treat debug and release builds 
as part of the normal development process. The only confusion I've seen 
has been related to CPython's not-quite-Windows-ish approach to debug 
builds, and in practically every case it's been enough to explain 
"release CPython uses a different CRT to your debug extension, but once 
you align those it'll be fine".

I definitely *do not* want to force or encourage package developers to 
release debug ABI versions of their prebuilt packages. But at the same 
time I don't want to remove the benefits that debug builds currently 
include.

Basically, I'm happy with the status quo, and the users I talk to are 
happy with it. So I'd rather not worry about optimising debug builds for 
speed or memory usage. (It's a question of direction more than anything 
else, and until we get some official statement of direction then I'll 
keep advocating a direction based on my experiences ;) )

Cheers,
Steve