From stefan_ml at behnel.de Sun Oct 29 17:06:36 2023 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Oct 2023 22:06:36 +0100 Subject: [Cython] Should we start using the internal CPython APIs? Message-ID: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> Hi all, given the latest blow against exposing implementation details of CPython in their C-API (see https://github.com/cython/cython/pull/5767 for the endless story), I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal This would give us greater freedom in accessing all the implementation details, so that we could directly integrate with those. We'd obviously still need one or more fallback implementations for "stable CPython", Limited API, PyPy and friends. There's a risk, clearly, that these internals change even during point releases. Maybe not a big risk, but not impossible either. We'd have to deal with that and so would our users. OTOH, having a single macro switch would make it easy for users to adapt if something breaks on their side, and also easy to benchmark if it makes a difference for their code. We could also leave it off by default and simply allow users with high performance needs to enable it manually. Or start by leaving it off until a new CPython X.Y release has stabilised and its (used-by-us) internals have proven not to change, and then switch it on for that release series. In any case, having a single switch for this feels like it could be easy to handle. What do you think? Stefan From dalcinl at gmail.com Mon Oct 30 01:59:25 2023 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 30 Oct 2023 08:59:25 +0300 Subject: [Cython] Should we start using the internal CPython APIs? In-Reply-To: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> References: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> Message-ID: On Mon, 30 Oct 2023 at 00:12, Stefan Behnel wrote: > Hi all, > > given the latest blow against exposing implementation details of CPython > in > their C-API (see https://github.com/cython/cython/pull/5767 for the > endless > story), In this new world order of political correctness, they will not say it directly, but the fact that Cython is probably the biggest consumer of these private APIs, this is a way to hard-press Cython to stop doing that once and for all. I remember some exchanges in GitHub where Python core devs manifested quite a bit of dislike/disdain for what Cython is doing. > I seriously start wondering if we shouldn't just define > "Py_BUILD_CORE" This sounds just too much, and it's like a war declaration on what they are trying to do. > (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro > guard that triggers its #define) and include the internal "pycore_*.h" > CPython header files from here: > > https://github.com/python/cpython/tree/main/Include/internal > > and this is totally reasonable, as it is under Cython's control. However, how much would this be different from CYTHON_LIMITED_API, or how the new define would interact with CYTHON_LIMITED_API? > > There's a risk, clearly, that these internals change even during point > releases. Maybe not a big risk, but not impossible either. We'd have to > deal with that and so would our users. > And that would be the biggest blow, actually. If this is truly a possibility, then I will have to turn off the use of CPython internals in my project. I don't want my already released packages to stop building just after a CPython patch release. There is no way I'm willing to deal with this mess, I already have too much maintenance work going on. > OTOH, having a single macro switch would make it easy for users to adapt > if > something breaks on their side, and also easy to benchmark if it makes a > difference for their code. > Definitely > > We could also leave it off by default and simply allow users with high > performance needs to enable it manually. Or start by leaving it off until > a > new CPython X.Y release has stabilised and its (used-by-us) internals have > proven not to change, and then switch it on for that release series. In > any > case, having a single switch for this feels like it could be easy to > handle. > Given that there is no guarantee of stability across point releases, I would argue that the macro should be off by default. Power users and packagers (that is, people that usually know how things work) can easily turn it on with "CFLAGS=-DFOO=1". What you do not want is novices and occasional users with little or no knowledge of Cython or even Python to flood your issue trackers with issues titled "I cannot install your thing" after a pip install failure. What about adding a Cython compiler directive to set the default value of the new macro? That way, it is up to the project's authors/maintainers to decide on how to handle this, they are in a better position to weigh the performance vs. portability trade-off. You still have to decide on the default value if the compiler directive is not set, but in this case I call for Stefan to pick it according to his preference/convenience, he certainly deserves the right to do so. -- Lisandro Dalcin ============ Senior Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matusvalo at gmail.com Mon Oct 30 14:27:46 2023 From: matusvalo at gmail.com (matus valo) Date: Mon, 30 Oct 2023 19:27:46 +0100 Subject: [Cython] Should we start using the internal CPython APIs? In-Reply-To: References: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> Message-ID: Hi All, > >> I seriously start wondering if we shouldn't just define >> "Py_BUILD_CORE" > > > This sounds just too much, and it's like a war declaration on what they > are trying to do. > I have the same opinion. I think this will negate all effort that the C/API workgroup is trying to do - to limit leaking internals of cpython. I would do this if there is no other way to move forward. I would suggest convincing cpython core devs to move API sharing cpython internals to unstable API tier [1] and to use it. Reasons: 1. This tier is intended for projects like cython ("Tools requiring access to CPython internals (e.g. advanced debuggers and JIT compilers)" [2]) hence I think there won't be huge opposition to provide needed low-level API for Cython. 2. Unstable tier has guaranteed stability for each major release so we will avoid issues with changes in minor releases which can happen for internal API Please keep in mind that I am not involved in python API in any way so this is just my personal opinion with very limited knowledge. Matus [1] https://docs.python.org/3/c-api/stable.html#unstable-c-api [2] https://peps.python.org/pep-0689/#unstable-c-api-tier -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Mon Oct 30 16:42:20 2023 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 30 Oct 2023 21:42:20 +0100 Subject: [Cython] Should we start using the internal CPython APIs? In-Reply-To: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> References: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> Message-ID: <504e5830-f95f-46ab-b7af-c043acec48a7@behnel.de> Thank you for your comments so far. Stefan Behnel schrieb am 29.10.23 um 22:06: > I seriously start wondering if we shouldn't just define > "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro > guard that triggers its #define) and include the internal "pycore_*.h" > CPython header files from here: > > https://github.com/python/cpython/tree/main/Include/internal I just remembered that there's a one major technical issue with this. CPython now requires C99 for its own code base (Py3.13 actually uses "-std=c11" on my side). While they care about keeping public header files compatible with C89 and C++, their internal header files may not always have that quality, and won't be tested for it. So, governance is one argument, but technical reasons can also make this appear less appealing overall. I'll let things settle some more and see in what direction Py3.13 will eventually be moving. Stefan