From dw-git at d-woods.co.uk Sat Nov 4 09:45:22 2023 From: dw-git at d-woods.co.uk (da-woods) Date: Sat, 4 Nov 2023 13:45:22 +0000 Subject: [Cython] Should we start using the internal CPython APIs? In-Reply-To: <504e5830-f95f-46ab-b7af-c043acec48a7@behnel.de> References: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> <504e5830-f95f-46ab-b7af-c043acec48a7@behnel.de> Message-ID: I'm a bit late in replying to this but here are some unordered thoughts. * I'm fairly relaxed about using `Py_BUILD_CORE` if useful - I think we mostly do have good fallback paths for most things so can adapt quickly when stuff changes. * CYTHON_USE_CPYTHON_CORE_DETAILS sounds reasonable, but it's yet another variation to test. * I wonder if fixing up the limited API implementation should be higher priority than creating a third level been "full" and "limited API". * I recall we were planning to ditch c89 as a strict requirement after 3.0? Incompatibility with C++ might be more of an issue though. * Even so, if there's a good way of turning it off then we could say: "if you want strict c89 support then you can't use CYTHON_USE_CPYTHON_CORE_DETAILS" and people would always have options. * Waiting and seeing may be a good option for now. David On 30/10/2023 20:42, Stefan Behnel wrote: > Thank you for your comments so far. > > Stefan Behnel schrieb am 29.10.23 um 22:06: >> I seriously start wondering if we shouldn't just define >> "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" >> macro guard that triggers its #define) and include the internal >> "pycore_*.h" CPython header files from here: >> >> https://github.com/python/cpython/tree/main/Include/internal > > I just remembered that there's a one major technical issue with this. > CPython now requires C99 for its own code base (Py3.13 actually uses > "-std=c11" on my side). While they care about keeping public header > files compatible with C89 and C++, their internal header files may not > always have that quality, and won't be tested for it. > > So, governance is one argument, but technical reasons can also make > this appear less appealing overall. > > I'll let things settle some more and see in what direction Py3.13 will > eventually be moving. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Sun Nov 5 17:06:50 2023 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 5 Nov 2023 23:06:50 +0100 Subject: [Cython] Ready for Cython 3.1 ? Message-ID: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> Hi all, it looks like Cython 3.0.6 is going to be a "most things fixed" kind of release for the 3.0.x release series. Given the work that lies ahead of us for Cython 3.1, I think we're at a point to get started on that, making the future 3.0.x releases stable and "boring". As a reminder, Cython 3.1 will remove support for Python 2.7 and Python 3.[567], i.e. all Python versions that are now EOL. Python 3.8 will continue to receive security fixes for another year. Python 3.7 is EOL but still up for debate since it's probably not hard to support and still maintained in some Linux distributions for another couple of years. But I'm fine with considering it legacy. We'll probably notice if it gets in the way while preparing Cython 3.0, and can leave support in until there's a reason to remove it. https://github.com/cython/cython/issues/2800 I'd like to ease our feature development by using more modern Python features in our code base and by targeting less Python versions in Cython 3.1 compared to the "all things supported" Cython 3.0. I also consider Cython 3.1 a prime target for better Limited API support. Users probably won't care both for that and for outdated Python versions at the same time. Or, they can use Cython 3.0.x for continued legacy support. Since Cython 3.1 is mostly about ripping out old code, we can try to keep the development cycle short, so that new features don't have to wait that long. Certainly not as long as for Cython 3.0? Is everyone and everything ready to start working on Cython 3.1? Stefan From stefan_ml at behnel.de Sun Nov 5 17:16:20 2023 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 5 Nov 2023 23:16:20 +0100 Subject: [Cython] Should we start using the internal CPython APIs? In-Reply-To: References: <218e8f3c-f77e-4106-ada7-5f52fc2bdce1@behnel.de> <504e5830-f95f-46ab-b7af-c043acec48a7@behnel.de> Message-ID: da-woods schrieb am 04.11.23 um 14:45: > I'm a bit late in replying to this but here are some unordered thoughts. > > * I'm fairly relaxed about using `Py_BUILD_CORE` if useful - I think we > mostly do have good fallback paths for most things so can adapt quickly > when stuff changes. I'm not entirely relaxed about it, but I agree that the fallbacks should usually make it easy to keep things working also after larger changes in CPython. > * CYTHON_USE_CPYTHON_CORE_DETAILS sounds reasonable, but it's yet another > variation to test. True. > * I wonder if fixing up the limited API implementation should be higher > priority than creating a third level been "full" and "limited API". I think there's potential for all three. Basically modes "aggressively fast", "highly compatible" and "version independent". The latter is what the Stable ABI together with the Limited API should give you. > * I recall we were planning to ditch c89 as a strict requirement after 3.0? > Incompatibility with C++ might be more of an issue though. Yes. C++ is not an issue for CPython, so their internal header files are not tested with C++ at all. That's the highest potential for breakage, if we accept to generate C99 from Cython 3.1 onwards. We should make sure that we use "-std=c89" in at least one Cython 3.0 test setup, BTW. > * Even so, if there's a good way of turning it off then we could say: "if > you want strict c89 support then you can't use > CYTHON_USE_CPYTHON_CORE_DETAILS" and people would always have options. That could be part of it, yes. > * Waiting and seeing may be a good option for now. I agree. This still seems best for now, especially given the amount of recent changes in the C-API. Let's wait for those to settle down, at least. Thanks everyone for your opinions and comments! Stefan From dalcinl at gmail.com Mon Nov 6 03:05:43 2023 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 6 Nov 2023 11:05:43 +0300 Subject: [Cython] Ready for Cython 3.1 ? In-Reply-To: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> References: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> Message-ID: On Mon, 6 Nov 2023 at 01:19, Stefan Behnel wrote: > Hi all, > > it looks like Cython 3.0.6 is going to be a "most things fixed" kind of > release for the 3.0.x release series. I'm having issues using CYTHON_LIMITED_API with some Python versions (<=3.9). If you are not in a rush to release 3.0.6, I would like to have some time to properly investigate what's going on. -- Lisandro Dalcin ============ Senior Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw-git at d-woods.co.uk Mon Nov 6 02:48:23 2023 From: dw-git at d-woods.co.uk (da-woods) Date: Mon, 6 Nov 2023 07:48:23 +0000 Subject: [Cython] Ready for Cython 3.1 ? In-Reply-To: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> References: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> Message-ID: <7775907c-3a2a-4d70-acbe-80b8246b52ae@d-woods.co.uk> I agree with most of this. > I also consider Cython 3.1 a prime target for better Limited API support. Yes - but I wouldn't treat complete support as a blocker (I don't think this is what you meant though). There's a separate question about what we consider the minimum viable Limited API version we want to support. I imagine that'll ultimately be decided by "what we can make work", but I don't think it'll be less that 3.4 (when PyType_GetSlot) was added. It's probably something to decide later. David On 05/11/2023 22:06, Stefan Behnel wrote: > Hi all, > > it looks like Cython 3.0.6 is going to be a "most things fixed" kind > of release for the 3.0.x release series. Given the work that lies > ahead of us for Cython 3.1, I think we're at a point to get started on > that, making the future 3.0.x releases stable and "boring". > > As a reminder, Cython 3.1 will remove support for Python 2.7 and > Python 3.[567], i.e. all Python versions that are now EOL. Python 3.8 > will continue to receive security fixes for another year. Python 3.7 > is EOL but still up for debate since it's probably not hard to support > and still maintained in some Linux distributions for another couple of > years. But I'm fine with considering it legacy. We'll probably notice > if it gets in the way while preparing Cython 3.0, and can leave > support in until there's a reason to remove it. > > https://github.com/cython/cython/issues/2800 > > I'd like to ease our feature development by using more modern Python > features in our code base and by targeting less Python versions in > Cython 3.1 compared to the "all things supported" Cython 3.0. > > I also consider Cython 3.1 a prime target for better Limited API > support. Users probably won't care both for that and for outdated > Python versions at the same time. Or, they can use Cython 3.0.x for > continued legacy support. > > Since Cython 3.1 is mostly about ripping out old code, we can try to > keep the development cycle short, so that new features don't have to > wait that long. Certainly not as long as for Cython 3.0? > > Is everyone and everything ready to start working on Cython 3.1? > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Mon Nov 6 04:46:29 2023 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 6 Nov 2023 10:46:29 +0100 Subject: [Cython] Ready for Cython 3.1 ? In-Reply-To: <7775907c-3a2a-4d70-acbe-80b8246b52ae@d-woods.co.uk> References: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> <7775907c-3a2a-4d70-acbe-80b8246b52ae@d-woods.co.uk> Message-ID: da-woods schrieb am 06.11.23 um 08:48: > > I also consider Cython 3.1 a prime target for better Limited API support. > > Yes - but I wouldn't treat complete support as a blocker (I don't think > this is what you meant though). It's experimental in 3.0 and I don't expect it to "fully" work in 3.1. > There's a separate question about what we consider the minimum viable > Limited API version we want to support. I imagine that'll ultimately be > decided by "what we can make work", but I don't think it'll be less that > 3.4 (when PyType_GetSlot) was added. It's probably something to decide later. That's another thing that moving the support to 3.1 would solve. If we can target Py3.7/3.8+ instead of older versions, then also the Limited API will be more usable. Stefan From stefan_ml at behnel.de Mon Nov 6 04:48:16 2023 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 6 Nov 2023 10:48:16 +0100 Subject: [Cython] Ready for Cython 3.1 ? In-Reply-To: References: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> Message-ID: Lisandro Dalcin schrieb am 06.11.23 um 09:05: > On Mon, 6 Nov 2023 at 01:19, Stefan Behnel wrote: >> it looks like Cython 3.0.6 is going to be a "most things fixed" kind of >> release for the 3.0.x release series. > > I'm having issues using CYTHON_LIMITED_API with some Python versions > (<=3.9). > If you are not in a rush to release 3.0.6, I would like to have some time > to properly investigate what's going on. I'd rather postpone these things to 3.1. They are not critical for 3.0, and as I wrote, I think it's actually helpful for users to target 3.1 rather than 3.0. Stefan From stefan_ml at behnel.de Mon Nov 6 07:53:12 2023 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 6 Nov 2023 13:53:12 +0100 Subject: [Cython] Ready for Cython 3.1 ? In-Reply-To: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> References: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> Message-ID: Stefan Behnel schrieb am 05.11.23 um 23:06: > I'd like to ease our feature development by using more modern Python > features in our code base and by targeting less Python versions in Cython > 3.1 compared to the "all things supported" Cython 3.0. I created a 3.0.x maintenance branch and removed the Py<3.7 test jobs from the master branch. That should make the CI response visibly faster. Happy code cleaning :) Stefan From matusvalo at gmail.com Tue Nov 7 10:39:09 2023 From: matusvalo at gmail.com (matus valo) Date: Tue, 7 Nov 2023 16:39:09 +0100 Subject: [Cython] Ready for Cython 3.1 ? In-Reply-To: References: <3f348012-3bed-4d81-ba41-84153886a0fb@behnel.de> Message-ID: Hi all, There are several PRs targeted to Cython 3.0.X [1]. Is it possible to push all of them or some of them are not ready? At least https://github.com/cython/cython/pull/5754 we can kick of without any doubt. Me personally, I would prefer to include following two PRs: * https://github.com/cython/cython/pull/5785 * https://github.com/cython/cython/pull/5233 Matus [1] https://github.com/cython/cython/pulls?q=is%3Aopen+is%3Apr+milestone%3A3.0.x On Mon, 6 Nov 2023 at 13:56, Stefan Behnel wrote: > Stefan Behnel schrieb am 05.11.23 um 23:06: > > I'd like to ease our feature development by using more modern Python > > features in our code base and by targeting less Python versions in > Cython > > 3.1 compared to the "all things supported" Cython 3.0. > > I created a 3.0.x maintenance branch and removed the Py<3.7 test jobs from > the master branch. That should make the CI response visibly faster. > > Happy code cleaning :) > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: