[Cython] Should we start using the internal CPython APIs?

[Lisandro Dalcin]

On Mon, 30 Oct 2023 at 00:12, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Hi all,
>
> given the latest blow against exposing implementation details of CPython
> in
> their C-API (see https://github.com/cython/cython/pull/5767 for the
> endless
> story),


In this new world order of political correctness, they will not say it
directly, but the fact that Cython is probably the biggest consumer of
these private APIs, this is a way to hard-press Cython to stop doing that
once and for all. I remember some exchanges in GitHub where Python core
devs manifested quite a bit of dislike/disdain for what Cython is doing.


> I seriously start wondering if we shouldn't just define
> "Py_BUILD_CORE"


This sounds just too much, and it's like a war declaration on what they are
trying to do.


> (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro
> guard that triggers its #define) and include the internal "pycore_*.h"
> CPython header files from here:
>
> https://github.com/python/cpython/tree/main/Include/internal
>
>
and this is totally reasonable, as it is under Cython's control.

However, how much would this be different from CYTHON_LIMITED_API, or how
the new define would interact with CYTHON_LIMITED_API?


>
> There's a risk, clearly, that these internals change even during point
> releases. Maybe not a big risk, but not impossible either. We'd have to
> deal with that and so would our users.
>

And that would be the biggest blow, actually. If this is truly a
possibility, then I will have to turn off the use of CPython internals in
my project. I don't want my already released packages to stop building
just after a CPython patch release. There is no way I'm willing to deal
with this mess, I already have too much maintenance work going on.


> OTOH, having a single macro switch would make it easy for users to adapt
> if
> something breaks on their side, and also easy to benchmark if it makes a
> difference for their code.
>

Definitely


>
> We could also leave it off by default and simply allow users with high
> performance needs to enable it manually. Or start by leaving it off until
> a
> new CPython X.Y release has stabilised and its (used-by-us) internals have
> proven not to change, and then switch it on for that release series. In
> any
> case, having a single switch for this feels like it could be easy to
> handle.
>

Given that there is no guarantee of stability across point releases, I
would argue that the macro should be off by default. Power users and
packagers (that is, people that usually know how things work) can easily
turn it on with "CFLAGS=-DFOO=1". What you do not want is novices and
occasional users with little or no knowledge of Cython or even Python to
flood your issue trackers with issues titled "I cannot install your thing"
after a pip install failure.

What about adding a Cython compiler directive to set the default value of
the new macro? That way, it is up to the project's authors/maintainers to
decide on how to handle this, they are in a better position to weigh the
performance vs. portability trade-off. You still have to decide on the
default value if the compiler directive is not set, but in this case I call
for Stefan to pick it according to his preference/convenience, he certainly
deserves the right to do so.


-- 
Lisandro Dalcin
============
Senior Research Scientist
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/cython-devel/attachments/20231030/15ccd730/attachment.html>