[Cython] Should we start using the internal CPython APIs?

[Stefan Behnel]

Hi all,

given the latest blow against exposing implementation details of CPython in 
their C-API (see https://github.com/cython/cython/pull/5767 for the endless 
story), I seriously start wondering if we shouldn't just define 
"Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro 
guard that triggers its #define) and include the internal "pycore_*.h" 
CPython header files from here:

https://github.com/python/cpython/tree/main/Include/internal

This would give us greater freedom in accessing all the implementation 
details, so that we could directly integrate with those. We'd obviously 
still need one or more fallback implementations for "stable CPython", 
Limited API, PyPy and friends.

There's a risk, clearly, that these internals change even during point 
releases. Maybe not a big risk, but not impossible either. We'd have to 
deal with that and so would our users.

OTOH, having a single macro switch would make it easy for users to adapt if 
something breaks on their side, and also easy to benchmark if it makes a 
difference for their code.

We could also leave it off by default and simply allow users with high 
performance needs to enable it manually. Or start by leaving it off until a 
new CPython X.Y release has stabilised and its (used-by-us) internals have 
proven not to change, and then switch it on for that release series. In any 
case, having a single switch for this feels like it could be easy to handle.

What do you think?

Stefan