From stefan_ml at behnel.de Thu Nov 1 22:22:54 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 01 Nov 2012 22:22:54 +0100 Subject: [Cython] Feedback from PyCon-DE Message-ID: <5092E82E.3060500@behnel.de> Hi, just wanted to mention that I met many happy Cython users at the conference. Some of them came to me after my talks to say "thanks!" for this great tool and enthusiastically told me how much work it saved them. I'm warmly forwarding this to all of you who made Cython what it is today. Stefan From stefan_ml at behnel.de Sun Nov 4 00:22:24 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 04 Nov 2012 00:22:24 +0100 Subject: [Cython] Any more fixes for 0.17.2? Message-ID: <5095A730.9050007@behnel.de> Hi, the 0.17 branch has gained a couple of fixes, so I'd like to release a 0.17.2 some time next week. In case you have other things to add to it within the next few days, please do, or send a quick reply if you'd like to have more time. Stefan From robertwb at gmail.com Sun Nov 4 02:23:38 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 3 Nov 2012 18:23:38 -0700 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <5095A730.9050007@behnel.de> References: <5095A730.9050007@behnel.de> Message-ID: On Nov 3, 2012 4:22 PM, "Stefan Behnel" wrote: > > Hi, > > the 0.17 branch has gained a couple of fixes, so I'd like to release a 0.17.2 some time next week. You read my mind. > In case you have other things to add to it within the next few days, please do, or send a quick reply if you'd like to have more time. > I've got a couple more tiny fixes, but next week is a reasonable timeframe. _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sun Nov 4 10:28:30 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 04 Nov 2012 10:28:30 +0100 Subject: [Cython] new changelog file in the source tree Message-ID: <5096353E.9010505@behnel.de> Hi, I added a CHANGES.rst file now that collects the changes for each release. Please add to it as you see fit. The content is reversed engineered from the Wiki pages. It would be good if others could give it a glance - I'm sure there are things to improve. Stefan From markflorisson88 at gmail.com Sun Nov 4 13:00:01 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 4 Nov 2012 12:00:01 +0000 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: References: <5095A730.9050007@behnel.de> Message-ID: On 4 November 2012 01:23, Robert Bradshaw wrote: > > On Nov 3, 2012 4:22 PM, "Stefan Behnel" wrote: >> >> Hi, >> >> the 0.17 branch has gained a couple of fixes, so I'd like to release a >> 0.17.2 some time next week. > > You read my mind. > >> In case you have other things to add to it within the next few days, >> please do, or send a quick reply if you'd like to have more time. >> > > I've got a couple more tiny fixes, but next week is a reasonable timeframe. > > _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > I have some fixes for fused types, and a fix that allows you to specify memoryviews from python, like cython.double[:, :], which allows you to specialize fused functions. From stefan_ml at behnel.de Sun Nov 4 13:04:34 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 04 Nov 2012 13:04:34 +0100 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: References: <5095A730.9050007@behnel.de>

Message-ID: <509659D2.6050605@behnel.de> mark florisson, 04.11.2012 13:00: >> On Nov 3, 2012 4:22 PM, "Stefan Behnel" wrote: >>> the 0.17 branch has gained a couple of fixes, so I'd like to release a >>> 0.17.2 some time next week. >> > >>> In case you have other things to add to it within the next few days, >>> please do, or send a quick reply if you'd like to have more time. > > I have some fixes for fused types, and a fix that allows you to > specify memoryviews from python, like cython.double[:, :], which > allows you to specialize fused functions. Very cool. If it's ready, I'd like to see that in. Stefan From markflorisson88 at gmail.com Sun Nov 4 13:45:23 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 4 Nov 2012 12:45:23 +0000 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <509659D2.6050605@behnel.de> References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> Message-ID: On 4 November 2012 12:04, Stefan Behnel wrote: > mark florisson, 04.11.2012 13:00: >>> >>> On Nov 3, 2012 4:22 PM, "Stefan Behnel" wrote: >>>> >>>> the 0.17 branch has gained a couple of fixes, so I'd like to release a >>>> 0.17.2 some time next week. >>> >>> > >>>> >>>> In case you have other things to add to it within the next few days, >>>> >>>> please do, or send a quick reply if you'd like to have more time. >> >> >> I have some fixes for fused types, and a fix that allows you to >> specify memoryviews from python, like cython.double[:, :], which >> allows you to specialize fused functions. > > > Very cool. If it's ready, I'd like to see that in. > > Stefan > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Great, I'm on it. It seems that some tests are failing in cython-devel since commit 'disable some Cython syntax in .py compilation mode: typecasts, &..., sizeof()', but you're probably aware of that. From stefan_ml at behnel.de Sun Nov 4 13:56:46 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 04 Nov 2012 13:56:46 +0100 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> Message-ID: <5096660E.4040806@behnel.de> mark florisson, 04.11.2012 13:45: > It seems that some tests are failing in cython-devel > since commit 'disable some Cython syntax in .py compilation mode: > typecasts, &..., sizeof()', but you're probably aware of that. Erm, no? Which ones? That change is also in cython-release, which tests cleanly. I had only seen that the overflow tests fail that Robert added. Stefan From markflorisson88 at gmail.com Sun Nov 4 14:07:04 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 4 Nov 2012 13:07:04 +0000 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <5096660E.4040806@behnel.de> References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> <5096660E.4040806@behnel.de> Message-ID: On 4 November 2012 12:56, Stefan Behnel wrote: > mark florisson, 04.11.2012 13:45: > >> It seems that some tests are failing in cython-devel >> since commit 'disable some Cython syntax in .py compilation mode: >> typecasts, &..., sizeof()', but you're probably aware of that. > > > Erm, no? Which ones? > > That change is also in cython-release, which tests cleanly. > > I had only seen that the overflow tests fail that Robert added. > > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Oh it should be fine then, it's the revision Jenkins pointed me to for the first failing test run (which needn't mean the commit is the fault of course). I also see a failing with gil test in py3km, maybe due to a change in exception printing or doctest itself. I'm surprised it doesn't fail in the release build. From markflorisson88 at gmail.com Sun Nov 4 21:42:17 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 4 Nov 2012 20:42:17 +0000 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <509659D2.6050605@behnel.de> References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> Message-ID: On 4 November 2012 12:04, Stefan Behnel wrote: > > mark florisson, 04.11.2012 13:00: >>> >>> On Nov 3, 2012 4:22 PM, "Stefan Behnel" wrote: >>>> >>>> the 0.17 branch has gained a couple of fixes, so I'd like to release a >>>> 0.17.2 some time next week. >>> >>> > >>>> >>>> In case you have other things to add to it within the next few days, >>>> >>>> please do, or send a quick reply if you'd like to have more time. >> >> >> I have some fixes for fused types, and a fix that allows you to >> specify memoryviews from python, like cython.double[:, :], which >> allows you to specialize fused functions. > > > Very cool. If it's ready, I'd like to see that in. I merged it in master and 0.17. > Stefan > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From oparcollet.triqs at gmail.com Sun Nov 4 23:32:55 2012 From: oparcollet.triqs at gmail.com (Olivier Parcollet) Date: Sun, 04 Nov 2012 23:32:55 +0100 Subject: [Cython] Custom automatic type conversions from/to C++ Message-ID: <5096ED17.4080107@gmail.com> Hello, I would like to write some automatic C++/python conversionsfor the classes of my project in addition to std::vector and others std:: objects. Conversions for std:: objects are written in Cython/Utility/CppConvert.pyx, i.e. an internal cython file. The question is then simply to load another CppConvert.pyx file that would be project dependent. Is there already a way to do it ? or a plan to do provide such user defined conversions ? I made a simple patch to load also another CppConvert.pyx from the current directory, which works for my code, but I would welcome a more general solution... Best, Olivier From markflorisson88 at gmail.com Mon Nov 5 12:02:43 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Mon, 5 Nov 2012 11:02:43 +0000 Subject: [Cython] Custom automatic type conversions from/to C++ In-Reply-To: <5096ED17.4080107@gmail.com> References: <5096ED17.4080107@gmail.com> Message-ID: On 4 November 2012 22:32, Olivier Parcollet wrote: > Hello, > > I would like to write some automatic C++/python conversionsfor the > classes of my project > in addition to std::vector and others std:: objects. > > Conversions for std:: objects are written in > Cython/Utility/CppConvert.pyx, i.e. an internal cython file. > > The question is then simply to load another CppConvert.pyx file that > would be project dependent. > Is there already a way to do it ? > or a plan to do provide such user defined conversions ? > > I made a simple patch to load also another CppConvert.pyx from the > current directory, > which works for my code, but I would welcome a more general solution... > Great. Ideally, Cython would be plugable, allowing one to use a documented API and to insert new visitors at certain points in the pipeline, allowing custom transformations and utilities. This is probably best addressed and proposed in the form of a CEP. You could then add a plugin that allows users to add C++ conversion rules. Some parts of the Cython code base may have to be adapted to allow them to be pluggable without monkey-patching Cython. A start could be just inserting new visitors and overriding C++ conversions, and leave the rest to the next person that wants to do more. > Best, > > Olivier > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Mon Nov 5 16:00:58 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 05 Nov 2012 16:00:58 +0100 Subject: [Cython] Custom automatic type conversions from/to C++ In-Reply-To: References: <5096ED17.4080107@gmail.com> Message-ID: <5097D4AA.80504@behnel.de> mark florisson, 05.11.2012 12:02: > On 4 November 2012 22:32, Olivier Parcollet wrote: >> I would like to write some automatic C++/python conversionsfor the >> classes of my project >> in addition to std::vector and others std:: objects. >> >> Conversions for std:: objects are written in >> Cython/Utility/CppConvert.pyx, i.e. an internal cython file. >> >> The question is then simply to load another CppConvert.pyx file that >> would be project dependent. >> Is there already a way to do it ? >> or a plan to do provide such user defined conversions ? >> >> I made a simple patch to load also another CppConvert.pyx from the >> current directory, >> which works for my code, but I would welcome a more general solution... >> > > Great. Ideally, Cython would be plugable, allowing one to use a > documented API and to insert new visitors at certain points in the > pipeline, allowing custom transformations and utilities I wouldn't call that "ideal". In case we want to support this kind of extensions, it should happen at the source code level, not hook into the compiler pipeline as part of the build process. IMHO, the right way to provide custom type conversions is by declaring them, e.g. as part of a C++ class declaration. BTW, I'm sure this has been discussed more than once on this list. Stefan From robertwb at gmail.com Mon Nov 5 20:49:47 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 5 Nov 2012 11:49:47 -0800 Subject: [Cython] Custom automatic type conversions from/to C++ In-Reply-To: <5097D4AA.80504@behnel.de> References: <5096ED17.4080107@gmail.com> <5097D4AA.80504@behnel.de> Message-ID: On Mon, Nov 5, 2012 at 7:00 AM, Stefan Behnel wrote: > mark florisson, 05.11.2012 12:02: >> On 4 November 2012 22:32, Olivier Parcollet wrote: >>> I would like to write some automatic C++/python conversionsfor the >>> classes of my project >>> in addition to std::vector and others std:: objects. >>> >>> Conversions for std:: objects are written in >>> Cython/Utility/CppConvert.pyx, i.e. an internal cython file. >>> >>> The question is then simply to load another CppConvert.pyx file that >>> would be project dependent. >>> Is there already a way to do it ? >>> or a plan to do provide such user defined conversions ? >>> >>> I made a simple patch to load also another CppConvert.pyx from the >>> current directory, >>> which works for my code, but I would welcome a more general solution... >>> >> >> Great. Ideally, Cython would be plugable, allowing one to use a >> documented API and to insert new visitors at certain points in the >> pipeline, allowing custom transformations and utilities > > I wouldn't call that "ideal". In case we want to support this kind of > extensions, it should happen at the source code level, not hook into the > compiler pipeline as part of the build process. > > IMHO, the right way to provide custom type conversions is by declaring > them, e.g. as part of a C++ class declaration. > > BTW, I'm sure this has been discussed more than once on this list. Yep. It's a question of complicating the language and coming up with the right syntax. For the stl containers, we decided to implement them directly and cross the generic bridge at some later point. Certainly worth writing up a proposal as a CEP http://wiki.cython.org/enhancements . - Robert From stefan_ml at behnel.de Tue Nov 6 20:22:44 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 06 Nov 2012 20:22:44 +0100 Subject: [Cython] porting mpi4py to PyPy runtime In-Reply-To: References: Message-ID: <50996384.1090301@behnel.de> Lisandro Dalcin, 02.10.2012 17:06: > After successfully porting mpi4py to PyPy 1.9, I want to share a few > issues I've encountered. The most serious onesare on PyPy's side, but > Cython do require a cuple of very minor fixes. > [...] > 4) mpi4py uses @classmethod decorator for many methods in cdef > classes. I really do not know what's going on, but method lookups do > not return the classmethod, but the original "unwrapped" method. I've > found a way to trick it, but again, I have no idea why it is working. > Basically, I have to execute hasattr() (in Python code, not Cython > code) on every class method: The hack reduces to adding the following > code at the very end of Cython code (note that I'm using "exec"): > > if PYPY: exec """ > def _pypy_setup(): > for klass in ( > Status, > Datatype, > Request, > Message, > Op, > Group, > Info, > Errhandler, > Comm, > Win, > File, > ): > for name in klass.__dict__: > meth = klass.__dict__[name] > if (isinstance(meth, classmethod) or > isinstance(meth, staticmethod)): > hasattr(klass, name) > _pypy_setup() > del _pypy_setup > """ > > I think (1) and (3) can be trivially handled in Cython, (3) is not an > issue for Cython as PyBuffer_FillInfo() is not directly used. And > about (4), we really need help from PyPy folks, they implement > PyType_Modified() as a non-op > (https://bitbucket.org/pypy/pypy/src/release-1.9/pypy/module/cpyext/typeobject.py#cl-726) > but that is not playing well with Cython. https://bugs.pypy.org/issue1106 I had already opened a ticket for this one a while ago. Stefan From stefan_ml at behnel.de Tue Nov 6 23:21:38 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 06 Nov 2012 23:21:38 +0100 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> <5096660E.4040806@behnel.de> Message-ID: <50998D72.8030800@behnel.de> mark florisson, 04.11.2012 14:07: > On 4 November 2012 12:56, Stefan Behnel wrote: >> mark florisson, 04.11.2012 13:45: >> >>> It seems that some tests are failing in cython-devel >>> since commit 'disable some Cython syntax in .py compilation mode: >>> typecasts, &..., sizeof()', but you're probably aware of that. >> >> Erm, no? Which ones? >> >> That change is also in cython-release, which tests cleanly. > > Oh it should be fine then, it's the revision Jenkins pointed me to for > the first failing test run (which needn't mean the commit is the fault > of course). I also see a failing with gil test in py3km, maybe due to > a change in exception printing or doctest itself. I'm surprised it > doesn't fail in the release build. Actually, our recent changes didn't end up on Jenkins due to a branch build configuration problem. They are up now and it seems to me that your changes have a problem with Py2.4 on the release branch. https://sage.math.washington.edu:8091/hudson/job/cython-release-tests/BACKEND=c,PYVERSION=py24-ext/50/testReport/doctest/DocTestCase/Doctest__numpy_test___test___test_dispatch_ndim/ Stefan From dalcinl at gmail.com Tue Nov 6 23:37:35 2012 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 6 Nov 2012 19:37:35 -0300 Subject: [Cython] porting mpi4py to PyPy runtime In-Reply-To: <50996384.1090301@behnel.de> References: <50996384.1090301@behnel.de> Message-ID: On 6 November 2012 16:22, Stefan Behnel wrote: > Lisandro Dalcin, 02.10.2012 17:06: >> After successfully porting mpi4py to PyPy 1.9, I want to share a few >> issues I've encountered. The most serious onesare on PyPy's side, but >> Cython do require a cuple of very minor fixes. >> [...] >> 4) mpi4py uses @classmethod decorator for many methods in cdef >> classes. I really do not know what's going on, but method lookups do >> not return the classmethod, but the original "unwrapped" method. I've >> found a way to trick it, but again, I have no idea why it is working. >> Basically, I have to execute hasattr() (in Python code, not Cython >> code) on every class method: The hack reduces to adding the following >> code at the very end of Cython code (note that I'm using "exec"): >> >> if PYPY: exec """ >> def _pypy_setup(): >> for klass in ( >> Status, >> Datatype, >> Request, >> Message, >> Op, >> Group, >> Info, >> Errhandler, >> Comm, >> Win, >> File, >> ): >> for name in klass.__dict__: >> meth = klass.__dict__[name] >> if (isinstance(meth, classmethod) or >> isinstance(meth, staticmethod)): >> hasattr(klass, name) >> _pypy_setup() >> del _pypy_setup >> """ >> >> I think (1) and (3) can be trivially handled in Cython, (3) is not an >> issue for Cython as PyBuffer_FillInfo() is not directly used. And >> about (4), we really need help from PyPy folks, they implement >> PyType_Modified() as a non-op >> (https://bitbucket.org/pypy/pypy/src/release-1.9/pypy/module/cpyext/typeobject.py#cl-726) >> but that is not playing well with Cython. > > https://bugs.pypy.org/issue1106 > > I had already opened a ticket for this one a while ago. > Can you test adding the following lines (add them after the Spam class) to see if them "fix" the issue? Such hint could be useful for PyPy folks: exec """ for name in Spam.__dict__: hasattr(Spam, name) """ -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From markflorisson88 at gmail.com Wed Nov 7 11:47:01 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 7 Nov 2012 10:47:01 +0000 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <50998D72.8030800@behnel.de> References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> <5096660E.4040806@behnel.de> <50998D72.8030800@behnel.de> Message-ID: On 6 November 2012 22:21, Stefan Behnel wrote: > mark florisson, 04.11.2012 14:07: >> On 4 November 2012 12:56, Stefan Behnel wrote: >>> mark florisson, 04.11.2012 13:45: >>> >>>> It seems that some tests are failing in cython-devel >>>> since commit 'disable some Cython syntax in .py compilation mode: >>>> typecasts, &..., sizeof()', but you're probably aware of that. >>> >>> Erm, no? Which ones? >>> >>> That change is also in cython-release, which tests cleanly. >> >> Oh it should be fine then, it's the revision Jenkins pointed me to for >> the first failing test run (which needn't mean the commit is the fault >> of course). I also see a failing with gil test in py3km, maybe due to >> a change in exception printing or doctest itself. I'm surprised it >> doesn't fail in the release build. > > Actually, our recent changes didn't end up on Jenkins due to a branch build > configuration problem. They are up now and it seems to me that your changes > have a problem with Py2.4 on the release branch. > > https://sage.math.washington.edu:8091/hudson/job/cython-release-tests/BACKEND=c,PYVERSION=py24-ext/50/testReport/doctest/DocTestCase/Doctest__numpy_test___test___test_dispatch_ndim/ > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel That's the issue where numpy headers define PyIndex_Check to 0 in python < 2.5. Maybe I should redefine it in the memoryview source... The same fix in my master branch seemed to pass though. From markflorisson88 at gmail.com Wed Nov 7 11:48:46 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 7 Nov 2012 10:48:46 +0000 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> <5096660E.4040806@behnel.de> <50998D72.8030800@behnel.de> Message-ID: On 7 November 2012 10:47, mark florisson wrote: > On 6 November 2012 22:21, Stefan Behnel wrote: >> mark florisson, 04.11.2012 14:07: >>> On 4 November 2012 12:56, Stefan Behnel wrote: >>>> mark florisson, 04.11.2012 13:45: >>>> >>>>> It seems that some tests are failing in cython-devel >>>>> since commit 'disable some Cython syntax in .py compilation mode: >>>>> typecasts, &..., sizeof()', but you're probably aware of that. >>>> >>>> Erm, no? Which ones? >>>> >>>> That change is also in cython-release, which tests cleanly. >>> >>> Oh it should be fine then, it's the revision Jenkins pointed me to for >>> the first failing test run (which needn't mean the commit is the fault >>> of course). I also see a failing with gil test in py3km, maybe due to >>> a change in exception printing or doctest itself. I'm surprised it >>> doesn't fail in the release build. >> >> Actually, our recent changes didn't end up on Jenkins due to a branch build >> configuration problem. They are up now and it seems to me that your changes >> have a problem with Py2.4 on the release branch. >> >> https://sage.math.washington.edu:8091/hudson/job/cython-release-tests/BACKEND=c,PYVERSION=py24-ext/50/testReport/doctest/DocTestCase/Doctest__numpy_test___test___test_dispatch_ndim/ >> >> Stefan >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > That's the issue where numpy headers define PyIndex_Check to 0 in > python < 2.5. Maybe I should redefine it in the memoryview source... > The same fix in my master branch seemed to pass though. But that test configuration always skips the numpy tests :). WHy is cython-devel blue then? From stefan_ml at behnel.de Thu Nov 8 13:52:30 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 08 Nov 2012 13:52:30 +0100 Subject: [Cython] Fwd: Make extension module initialisation more like Python module initialisation In-Reply-To: References: Message-ID: <509BAB0E.6050406@behnel.de> Hi, I just sent this to python-dev. Please comment over there if you have an opinion on this. http://thread.gmane.org/gmane.comp.python.devel/135764 Stefan -------- Original-Message -------- Hi, I suspect that this will be put into a proper PEP at some point, but I'd like to bring this up for discussion first. This came out of issues 13429 and 16392. http://bugs.python.org/issue13429 http://bugs.python.org/issue16392 Stefan The problem =========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it's not trivial to correctly add the module to sys.modules either. We specifically run into this for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. The proposal ============ I propose to split the extension module initialisation into two steps in Python 3.4, in a backwards compatible way. Step 1: The current module init function can be reduced to just creating the module instance and returning it (and potentially doing some simple C level setup). Optionally, after creating the module (and this is the new part), the module init code can register a C callback function that will be called after setting up the module. Step 2: The shared library importer receives the module instance from the module init function, adds __file__, __path__, __package__ and friends to the module dict, and then checks for the callback. If non-NULL, it calls it to continue the module initialisation by user code. The callback ============ The callback is defined as follows:: int (*PyModule_init_callback)(PyObject* the_module, PyModuleInitContext* context) "PyModuleInitContext" is a struct that is meant mostly for making the callback more future proof by allowing additional parameters to be passed in. For now, I can see a use case for the following fields:: struct PyModuleInitContext { char* module_name; char* qualified_module_name; } Both names are encoded in UTF-8. As for the file path, I consider it best to retrieve it from the module's __file__ attribute as a Python string object to reduce filename encoding problems. Note that this struct argument is not strictly required, but given that this proposal would have been much simpler if the module init function had accepted such an argument in the first place, I consider it a good idea not to let this chance pass by again. The registration of the callback uses a new C-API function: int PyModule_SetInitFunction(PyObject* module, PyModule_init_callback callback) The function name uses "Set" instead of "Register" to make it clear that there is only one such function per module. An alternative would be a new module creation function "PyModule_Create3()" that takes the callback as third argument, in addition to what "PyModule_Create2()" accepts. This would require users to explicitly pass in the (second) version argument, which might be considered only a minor issue. Implementation ============== The implementation requires local changes to the extension module importer and a new C-API function. In order to store the callback, it should use a new field in the module object struct. Open questions ============== It is not clear how extensions should be handled that register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case, although not the import file path. Alternatively, the import machinery could use a stack to remember for which modules a callback was registered during the last init function call, set up all of them and then call their callbacks. It's not clear if this meets the intention of the user. Alternatives ============ 1) It would be possible to make extension modules optionally export another symbol, e.g. "PyInit2_modulename", that the shared library loader would call in addition to the required function "PyInit_modulename". This would remove the need for a new API that registers the above callback. The drawback is that it also makes it easier to write broken code because a Python version or implementation that does not support this second symbol would simply not call it, without error. The new C-API function would let the build fail instead if it is not supported. 2) The callback could be made available as a Python function in the module dict, thus also removing the need for an explicit registration API. However, this approach would add overhead to both sides, the importer code and the user provided module init code, as it would require additional dictionary handling and the implementation of a one-time Python function in user code. It would also suffer from the problem that missing support in the runtime would pass silently. 3) The callback could be registered statically in the PyModuleDef struct by adding a new field. This is not trivial to do in a backwards compatible way because the struct would grow longer without explicit initialisation by existing user code. Extending PyModuleDef_HEAD_INIT might be possible but would still break at least binary compatibility. 4) Pass a new context argument into the module init function that contains all information necessary to properly and completely set up the module at creation time. This would provide a much simpler and cleaner solution than the proposed solution. However, it will not be possible before Python 4 as it breaks backwards compatibility with all existing extension modules at both the source and binary level. From stefan_ml at behnel.de Fri Nov 9 21:41:21 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 09 Nov 2012 21:41:21 +0100 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> <5096660E.4040806@behnel.de> <50998D72.8030800@behnel.de>

Message-ID: <509D6A71.7000104@behnel.de> mark florisson, 07.11.2012 11:48: > On 7 November 2012 10:47, mark florisson wrote: >> On 6 November 2012 22:21, Stefan Behnel wrote: >>> mark florisson, 04.11.2012 14:07: >>>> On 4 November 2012 12:56, Stefan Behnel wrote: >>>>> mark florisson, 04.11.2012 13:45: >>>>> >>>>>> It seems that some tests are failing in cython-devel >>>>>> since commit 'disable some Cython syntax in .py compilation mode: >>>>>> typecasts, &..., sizeof()', but you're probably aware of that. >>>>> >>>>> Erm, no? Which ones? >>>>> >>>>> That change is also in cython-release, which tests cleanly. >>>> >>>> Oh it should be fine then, it's the revision Jenkins pointed me to for >>>> the first failing test run (which needn't mean the commit is the fault >>>> of course). I also see a failing with gil test in py3km, maybe due to >>>> a change in exception printing or doctest itself. I'm surprised it >>>> doesn't fail in the release build. >>> >>> Actually, our recent changes didn't end up on Jenkins due to a branch build >>> configuration problem. They are up now and it seems to me that your changes >>> have a problem with Py2.4 on the release branch. >>> >>> https://sage.math.washington.edu:8091/hudson/job/cython-release-tests/BACKEND=c,PYVERSION=py24-ext/50/testReport/doctest/DocTestCase/Doctest__numpy_test___test___test_dispatch_ndim/ >> >> That's the issue where numpy headers define PyIndex_Check to 0 in >> python < 2.5. Maybe I should redefine it in the memoryview source... >> The same fix in my master branch seemed to pass though. > > But that test configuration always skips the numpy tests :). WHy is > cython-devel blue then? Did you notice that the failing test doesn't exist in the master? numpy_test.pyx doesn't have a "test_dispatch_ndim" there. Stefan From markflorisson88 at gmail.com Sun Nov 11 01:36:27 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 11 Nov 2012 00:36:27 +0000 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <509D6A71.7000104@behnel.de> References: <5095A730.9050007@behnel.de>

<509659D2.6050605@behnel.de> <5096660E.4040806@behnel.de> <50998D72.8030800@behnel.de>

<509D6A71.7000104@behnel.de> Message-ID: On 9 November 2012 20:41, Stefan Behnel wrote: > mark florisson, 07.11.2012 11:48: >> On 7 November 2012 10:47, mark florisson wrote: >>> On 6 November 2012 22:21, Stefan Behnel wrote: >>>> mark florisson, 04.11.2012 14:07: >>>>> On 4 November 2012 12:56, Stefan Behnel wrote: >>>>>> mark florisson, 04.11.2012 13:45: >>>>>> >>>>>>> It seems that some tests are failing in cython-devel >>>>>>> since commit 'disable some Cython syntax in .py compilation mode: >>>>>>> typecasts, &..., sizeof()', but you're probably aware of that. >>>>>> >>>>>> Erm, no? Which ones? >>>>>> >>>>>> That change is also in cython-release, which tests cleanly. >>>>> >>>>> Oh it should be fine then, it's the revision Jenkins pointed me to for >>>>> the first failing test run (which needn't mean the commit is the fault >>>>> of course). I also see a failing with gil test in py3km, maybe due to >>>>> a change in exception printing or doctest itself. I'm surprised it >>>>> doesn't fail in the release build. >>>> >>>> Actually, our recent changes didn't end up on Jenkins due to a branch build >>>> configuration problem. They are up now and it seems to me that your changes >>>> have a problem with Py2.4 on the release branch. >>>> >>>> https://sage.math.washington.edu:8091/hudson/job/cython-release-tests/BACKEND=c,PYVERSION=py24-ext/50/testReport/doctest/DocTestCase/Doctest__numpy_test___test___test_dispatch_ndim/ >>> >>> That's the issue where numpy headers define PyIndex_Check to 0 in >>> python < 2.5. Maybe I should redefine it in the memoryview source... >>> The same fix in my master branch seemed to pass though. >> >> But that test configuration always skips the numpy tests :). WHy is >> cython-devel blue then? > > Did you notice that the failing test doesn't exist in the master? > > numpy_test.pyx doesn't have a "test_dispatch_ndim" there. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Oh sorry, my mind had wandered off, I thought I merged it. Anyway, release should be blue after the test run. From stefan_ml at behnel.de Sun Nov 11 09:12:58 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 Nov 2012 09:12:58 +0100 Subject: [Cython] redefining PyIndex_Check Message-ID: <509F5E0A.5040809@behnel.de> Hi, instead of overriding an existing definition of "PyIndex_Check" like this: """ +#if PY_VERSION_HEX < 0x02050000 + /* NumPy headers define PyIndex_Check incorrectly */ + #undef PyIndex_Check + #define PyIndex_Check(o) (PyNumber_Check(o) && !PyFloat_Check(o) && !PyComplex_Check(o)) +#endif """ which may or may not have come from NumPy, shouldn't we be using our own definition of a "__Pyx_PyIndex_Check" in our code instead? Stefan From markflorisson88 at gmail.com Sun Nov 11 12:26:19 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 11 Nov 2012 11:26:19 +0000 Subject: [Cython] redefining PyIndex_Check In-Reply-To: <509F5E0A.5040809@behnel.de> References: <509F5E0A.5040809@behnel.de> Message-ID: On 11 November 2012 08:12, Stefan Behnel wrote: > Hi, > > instead of overriding an existing definition of "PyIndex_Check" like this: > > """ > +#if PY_VERSION_HEX < 0x02050000 > + /* NumPy headers define PyIndex_Check incorrectly */ > + #undef PyIndex_Check > + #define PyIndex_Check(o) (PyNumber_Check(o) && !PyFloat_Check(o) && > !PyComplex_Check(o)) > +#endif > """ > > which may or may not have come from NumPy, shouldn't we be using our own > definition of a "__Pyx_PyIndex_Check" in our code instead? > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel I thought about that, but then people do expect PyIndex_Check to work when they cimport it from cpython. But this fix doesn't really address that either, it just works if you use memoryviews (it's more of a quick and dirty hack, this fix needs to be associated with the cimport of numpy). Maybe we should use __Pyx_PyIndex_Check internally, and define PyIndex_Check like we do now for user convenience, and let numpy break it? Mark From stefan_ml at behnel.de Sun Nov 11 14:04:45 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 Nov 2012 14:04:45 +0100 Subject: [Cython] redefining PyIndex_Check In-Reply-To: References: <509F5E0A.5040809@behnel.de> Message-ID: <509FA26D.7010108@behnel.de> mark florisson, 11.11.2012 12:26: > On 11 November 2012 08:12, Stefan Behnel wrote: >> instead of overriding an existing definition of "PyIndex_Check" like this: >> >> """ >> +#if PY_VERSION_HEX < 0x02050000 >> + /* NumPy headers define PyIndex_Check incorrectly */ >> + #undef PyIndex_Check >> + #define PyIndex_Check(o) (PyNumber_Check(o) && !PyFloat_Check(o) && >> !PyComplex_Check(o)) >> +#endif >> """ >> >> which may or may not have come from NumPy, shouldn't we be using our own >> definition of a "__Pyx_PyIndex_Check" in our code instead? > > I thought about that, but then people do expect PyIndex_Check to work > when they cimport it from cpython. But this fix doesn't really address > that either, it just works if you use memoryviews (it's more of a > quick and dirty hack, this fix needs to be associated with the cimport > of numpy). > > Maybe we should use __Pyx_PyIndex_Check internally, and define > PyIndex_Check like we do now for user convenience, and let numpy break > it? +1 for all three points. Cython's own code should always be safe, users won't normally use that function anyway and NumPy would only break their code in Py2.4, which is sufficiently rarely used. BTW, has anyone brought this to the attention of the NumPy developers yet? They might want to fix their definition. Stefan From markflorisson88 at gmail.com Sun Nov 11 15:09:43 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 11 Nov 2012 14:09:43 +0000 Subject: [Cython] redefining PyIndex_Check In-Reply-To: <509FA26D.7010108@behnel.de> References: <509F5E0A.5040809@behnel.de> <509FA26D.7010108@behnel.de> Message-ID: On 11 November 2012 13:04, Stefan Behnel wrote: > mark florisson, 11.11.2012 12:26: >> On 11 November 2012 08:12, Stefan Behnel wrote: >>> instead of overriding an existing definition of "PyIndex_Check" like this: >>> >>> """ >>> +#if PY_VERSION_HEX < 0x02050000 >>> + /* NumPy headers define PyIndex_Check incorrectly */ >>> + #undef PyIndex_Check >>> + #define PyIndex_Check(o) (PyNumber_Check(o) && !PyFloat_Check(o) && >>> !PyComplex_Check(o)) >>> +#endif >>> """ >>> >>> which may or may not have come from NumPy, shouldn't we be using our own >>> definition of a "__Pyx_PyIndex_Check" in our code instead? >> >> I thought about that, but then people do expect PyIndex_Check to work >> when they cimport it from cpython. But this fix doesn't really address >> that either, it just works if you use memoryviews (it's more of a >> quick and dirty hack, this fix needs to be associated with the cimport >> of numpy). >> >> Maybe we should use __Pyx_PyIndex_Check internally, and define >> PyIndex_Check like we do now for user convenience, and let numpy break >> it? > > +1 for all three points. Cython's own code should always be safe, users > won't normally use that function anyway and NumPy would only break their > code in Py2.4, which is sufficiently rarely used. Great, I'm on it. On second consideration, it's better to not break it at all, and just define it to __Pyx_PyIndex_Check in the pxd. > BTW, has anyone brought this to the attention of the NumPy developers yet? > They might want to fix their definition. I haven't yet, maybe I'll just do a PR for such a trivial fix -- although maybe they have a reason for their definition. > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Sun Nov 11 15:51:48 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 11 Nov 2012 14:51:48 +0000 Subject: [Cython] redefining PyIndex_Check In-Reply-To: References: <509F5E0A.5040809@behnel.de> <509FA26D.7010108@behnel.de> Message-ID: On 11 November 2012 14:09, mark florisson wrote: > On 11 November 2012 13:04, Stefan Behnel wrote: >> mark florisson, 11.11.2012 12:26: >>> On 11 November 2012 08:12, Stefan Behnel wrote: >>>> instead of overriding an existing definition of "PyIndex_Check" like this: >>>> >>>> """ >>>> +#if PY_VERSION_HEX < 0x02050000 >>>> + /* NumPy headers define PyIndex_Check incorrectly */ >>>> + #undef PyIndex_Check >>>> + #define PyIndex_Check(o) (PyNumber_Check(o) && !PyFloat_Check(o) && >>>> !PyComplex_Check(o)) >>>> +#endif >>>> """ >>>> >>>> which may or may not have come from NumPy, shouldn't we be using our own >>>> definition of a "__Pyx_PyIndex_Check" in our code instead? >>> >>> I thought about that, but then people do expect PyIndex_Check to work >>> when they cimport it from cpython. But this fix doesn't really address >>> that either, it just works if you use memoryviews (it's more of a >>> quick and dirty hack, this fix needs to be associated with the cimport >>> of numpy). >>> >>> Maybe we should use __Pyx_PyIndex_Check internally, and define >>> PyIndex_Check like we do now for user convenience, and let numpy break >>> it? >> >> +1 for all three points. Cython's own code should always be safe, users >> won't normally use that function anyway and NumPy would only break their >> code in Py2.4, which is sufficiently rarely used. > > Great, I'm on it. On second consideration, it's better to not break it > at all, and just define it to __Pyx_PyIndex_Check in the pxd. > >> BTW, has anyone brought this to the attention of the NumPy developers yet? >> They might want to fix their definition. > > I haven't yet, maybe I'll just do a PR for such a trivial fix -- > although maybe they have a reason for their definition. Great, Pauli fixed it a while back: https://github.com/numpy/numpy/pull/307 >> Stefan >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Mon Nov 12 08:25:58 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 12 Nov 2012 08:25:58 +0100 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <5095A730.9050007@behnel.de> References: <5095A730.9050007@behnel.de> Message-ID: <50A0A486.9020800@behnel.de> Stefan Behnel, 04.11.2012 00:22: > the 0.17 branch has gained a couple of fixes, so I'd like to release a > 0.17.2 some time next week. In case you have other things to add to it > within the next few days, please do, or send a quick reply if you'd like to > have more time. Ok, the branch looks good. Any more changes before the release? Stefan From robertwb at gmail.com Mon Nov 12 20:35:25 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 12 Nov 2012 11:35:25 -0800 Subject: [Cython] Any more fixes for 0.17.2? In-Reply-To: <50A0A486.9020800@behnel.de> References: <5095A730.9050007@behnel.de> <50A0A486.9020800@behnel.de> Message-ID: On Sun, Nov 11, 2012 at 11:25 PM, Stefan Behnel wrote: > Stefan Behnel, 04.11.2012 00:22: >> the 0.17 branch has gained a couple of fixes, so I'd like to release a >> 0.17.2 some time next week. In case you have other things to add to it >> within the next few days, please do, or send a quick reply if you'd like to >> have more time. > > Ok, the branch looks good. Any more changes before the release? I say we cut a release candidate, and if there's no major hiccups we make it into a release. - Robert From stefan_ml at behnel.de Wed Nov 14 20:53:57 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 14 Nov 2012 20:53:57 +0100 Subject: [Cython] Release candidate 0.17.2rc1 Message-ID: <50A3F6D5.5080301@behnel.de> Hi, here is a release candidate for 0.17.2. http://cython.org/release/Cython-0.17.2rc1.tar.gz Please give it a try. If there are no (serious) regressions by the end of the week, this will be our next release. Stefan From stefan_ml at behnel.de Thu Nov 15 20:02:38 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 15 Nov 2012 20:02:38 +0100 Subject: [Cython] [cython-users] Re: Release candidate 0.17.2rc1 In-Reply-To: <5fea0717-7646-4883-8715-3d8af4d71818@googlegroups.com> References: <50A3F6D5.5080301@behnel.de> <5fea0717-7646-4883-8715-3d8af4d71818@googlegroups.com> Message-ID: <50A53C4E.6050605@behnel.de> Jasmine Sandhu, 15.11.2012 19:36: > Thanks for the heads up Stefan. > We are developing on the Mac and Windows and using Cython to wrap C++ code. > I tested the new Cython on windows against our code base. The setup and > install was easy. > > I build our application with the new Cython on Windows. It builds fine and > passes the unit tests. > No problems or issues thus far. Thanks for testing, especially on so many platforms! Stefan From gerald.dalley at gmail.com Fri Nov 16 17:54:51 2012 From: gerald.dalley at gmail.com (Gerald Dalley) Date: Fri, 16 Nov 2012 11:54:51 -0500 Subject: [Cython] incorrect codegen for nogil functions calling nogil except+ functions Message-ID: When a nogil function calls a nogil except+ function, Cython generates C code that requires the Py_BEGIN_ALLOW_THREADS macro to be evaluated, but it doesn't inject the code to evaluate the Py_BEING_ALLOW_THREADS macro. Consider the following C header file (nogil_except.h) inline int foo() { return 1; } and a corresponding Cython source file (nogil_except.pyx) cdef extern from "nogil_except.h": cdef int foo() nogil except+ cdef int bar() nogil: return foo() print bar() Cython's ExprNodes.py generates code that calls the Py_BLOCKS_THREADS macro, but it doesn't ensure that Py_BEGIN_ALLOW_THREADS has been injected into an enclosing scope. The latter macro defines a variable, "_save" that's needed by the Py_BLOCKS_THREADS macro: $ cython nogil_except.pyx && /usr/bin/g++ ... -c nogil_except.cpp nogil_except.c: In function ?int __pyx_f_12nogil_except_bar()?: nogil_except.c:537: error: ?_save? was not declared in this scope Here's the applicable block of generated code: $ awk -F, '{if (($2=="") && (NR >= 531) && (NR < 540)) printf "%-4d %s\n", NR,$0}' nogil_except.c 531 /* "nogil_except.pyx":4 532 * cdef int foo() nogil except+ 533 * cdef int bar() nogil: 534 * return foo() # <<<<<<<<<<<<<< 535 * print bar() 536 */ 537 try {__pyx_t_1 = foo();} catch(...) { Py_BLOCK_THREADS; __Pyx_CppExn2PyErr(); Py_UNBLOCK_THREADS; {__pyx_filename = __pyx_f[0]; __pyx_lineno = 4; __pyx_clineno = __LINE__; goto __pyx_L1_error;}} 538 __pyx_r = __pyx_t_1; 539 goto __pyx_L0; It seems like either ExprNodes.py should avoid injecting Py_BLOCK_THREADS and Py_UNBLOCK_THREADS when operating outside an OpenMP context and/or something should guarantee that Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS get injected as needed into the code stream. I've tested Cython 0.17b2 and 0.17b3 and both exhibit this problem. -- --Gerald Dalley dalleyg at ieee.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Nov 17 12:08:30 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Nov 2012 12:08:30 +0100 Subject: [Cython] [cython-users] Test case that will segfault In-Reply-To: References: Message-ID: <50A7702E.5000909@behnel.de> Hi, thanks for the test case. Ewald de Wit, 16.11.2012 23:09: > I have attached some code below that will lead to a segmentation fault on > Linux. The code iterates over a linked list of nodes and removes a node > while doing so. Adding an extra ref to the removed node will prevent the > crash, so the problem might be a refcount going to zero before its time. > This is with Cython 0.17.1 - not sure if bug or known limitation. > ------------------------------------------------------------------------------------------------------------------------- > cdef class Node: > > cdef public int value > cdef public Node nextNode > > def __init__(self, value): > self.value = value > self.nextNode = None > > > cdef class Chain: > > cdef public Node firstNode > > def __init__(self): > self.firstNode = None Note that Python object attributes are automatically initialised to None, so the above line is redundant. > def __iter__(self): > n = self.firstNode > while n is not None: > yield n > n = n.nextNode > > def add(self, Node node): > if self.firstNode is None: > self.firstNode = node > else: > for n in self: pass > n.nextNode = node > > def remove(self, Node node): > if node is self.firstNode: > self.firstNode = node.nextNode > else: > for n in self: > if node is n.nextNode: > n.nextNode = node.nextNode > break > > def dump(self): > for n in self: > print n.value > > > def test(): > c = Chain() > c.add(Node(1)) > c.add(Node(2)) > c.add(Node(3)) > c.add(Node(4)) > n = c.firstNode > while n is not None: > if n.value == 3: > c.remove(n) > #nn = n # Adding extra ref will prevent crash > n = n.nextNode # Crash will happen here at value 3 The C code generated for the last line is as follows: __Pyx_INCREF(((PyObject *)__pyx_v_n->nextNode)); __Pyx_DECREF(((PyObject *)__pyx_v_n)); __pyx_v_n = __pyx_v_n->nextNode; That's definitely unhealthy code. For a test case, the following would be enough: """ cdef class Node: cdef public Node next def test(): cdef Node n = Node() n.next = Node() n = n.next test() """ However, this doesn't crash for me, likely because the DECREF() above doesn't free the memory, just release it to CPython's memory allocator. So we're just lucky here. Stefan From stefan_ml at behnel.de Sat Nov 17 12:31:54 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Nov 2012 12:31:54 +0100 Subject: [Cython] [cython-users] Test case that will segfault In-Reply-To: <50A7702E.5000909@behnel.de> References: <50A7702E.5000909@behnel.de> Message-ID: <50A775AA.6030402@behnel.de> Stefan Behnel, 17.11.2012 12:08: > Ewald de Wit, 16.11.2012 23:09: >> I have attached some code below that will lead to a segmentation fault on >> Linux. The code iterates over a linked list of nodes and removes a node >> while doing so. Adding an extra ref to the removed node will prevent the >> crash, so the problem might be a refcount going to zero before its time. >> This is with Cython 0.17.1 - not sure if bug or known limitation. >> ------------------------------------------------------------------------------------------------------------------------- >> cdef class Node: >> >> cdef public int value >> cdef public Node nextNode >> >> def __init__(self, value): >> self.value = value >> self.nextNode = None >> >> >> cdef class Chain: >> >> cdef public Node firstNode >> >> def __init__(self): >> self.firstNode = None > > Note that Python object attributes are automatically initialised to None, > so the above line is redundant. > > >> def __iter__(self): >> n = self.firstNode >> while n is not None: >> yield n >> n = n.nextNode >> >> def add(self, Node node): >> if self.firstNode is None: >> self.firstNode = node >> else: >> for n in self: pass >> n.nextNode = node >> >> def remove(self, Node node): >> if node is self.firstNode: >> self.firstNode = node.nextNode >> else: >> for n in self: >> if node is n.nextNode: >> n.nextNode = node.nextNode >> break >> >> def dump(self): >> for n in self: >> print n.value >> >> >> def test(): >> c = Chain() >> c.add(Node(1)) >> c.add(Node(2)) >> c.add(Node(3)) >> c.add(Node(4)) >> n = c.firstNode >> while n is not None: >> if n.value == 3: >> c.remove(n) >> #nn = n # Adding extra ref will prevent crash >> n = n.nextNode # Crash will happen here at value 3 > > The C code generated for the last line is as follows: > > __Pyx_INCREF(((PyObject *)__pyx_v_n->nextNode)); > __Pyx_DECREF(((PyObject *)__pyx_v_n)); > __pyx_v_n = __pyx_v_n->nextNode; > > That's definitely unhealthy code. For a test case, the following would be > enough: > > """ > cdef class Node: > cdef public Node next > > def test(): > cdef Node n = Node() > n.next = Node() > n = n.next > > test() > """ > > However, this doesn't crash for me, likely because the DECREF() above > doesn't free the memory, just release it to CPython's memory allocator. So > we're just lucky here. Here is a proposed fix: https://github.com/cython/cython/commit/ea6a71acb5c79afb080855be1cb6ca30d283ec25 If Jenkins is happy with it, I'll add it to the release branch for 0.17.2. Stefan From L.J.Buitinck at uva.nl Tue Nov 20 00:06:37 2012 From: L.J.Buitinck at uva.nl (Lars Buitinck) Date: Tue, 20 Nov 2012 00:06:37 +0100 Subject: [Cython] bug report: compiler crash when assigning from function call returning ndarray Message-ID: Hi all, I just got a compiler crash (Cython 0.17, 0.17.1 and current master) when trying to compile a call to a function returning an np.ndarray: cimport numpy as np cdef np.ndarray[np.float64_t, ndim=2] trivial(X): return X def test(): # Uncommenting the following line, or removing the assignment, # prevents the crash: #cdef np.ndarray[np.float64_t, ndim=2] Z Z = trivial(np.zeros(4)) I'm aware that the program contains an error and it's not much of a hindrance to the project I'm working on, but the error message just isn't very friendly: Compiler crash traceback from this point on: File "/home/lars/src/cython/Cython/Compiler/ExprNodes.py", line 1528, in analyse_target_types Buffer.used_buffer_aux_vars(self.entry) File "/home/lars/src/cython/Cython/Compiler/Buffer.py", line 280, in used_buffer_aux_vars buffer_aux.buflocal_nd_var.used = True AttributeError: 'NoneType' object has no attribute 'buflocal_nd_var' More details at https://gist.github.com/4114677. I looked at Buffer.py and ExprNodes.py, but I couldn't figure out what exactly was going on. Could someone please look into this? TIA, -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam From stefan_ml at behnel.de Tue Nov 20 21:51:31 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 20 Nov 2012 21:51:31 +0100 Subject: [Cython] Cython 0.17.2 released Message-ID: <50ABED53.8070104@behnel.de> Hi everyone, I'm happy to announce the release of Cython 0.17.2. This is a (mostly) bug fix release for the stable 0.17 release series. http://pypi.python.org/pypi/Cython/0.17.2 Direct downloads: http://cython.org/release/Cython-0.17.2.tar.gz http://cython.org/release/Cython-0.17.2.zip Github version: https://github.com/cython/cython/commit/10183d61eb5cf33e6912dec2ab09740498f0947c Have fun, Stefan Complete changelog: 0.17.2 (2012-11-20) =================== Features added -------------- * ``cythonize()`` gained a best effort compile mode that can be used to simply ignore .py files that fail to compile. Bugs fixed ---------- * Replacing an object reference with the value of one of its cdef attributes could generate incorrect C code that accessed the object after deleting its last reference. * C-to-Python type coercions during cascaded comparisons could generate invalid C code, specifically when using the 'in' operator. * "obj[1,]" passed a single integer into the item getter instead of a tuple. * Cyclic imports at module init time did not work in Py3. * The names of C++ destructors for template classes were built incorrectly. * In pure mode, type casts in Cython syntax and the C ampersand operator are now rejected. Use the pure mode replacements instead. * In pure mode, C type names and the sizeof() function are no longer recognised as such and can be used as normal Python names. * The extended C level support for the CPython array type was declared too late to be used by user defined classes. * C++ class nesting was broken. * Better checking for required nullary constructors for stack-allocated C++ instances. * Remove module docstring in no-docstring mode. * Fix specialization for varargs function signatures. * Fix several compiler crashes. From matej at laitl.cz Fri Nov 30 01:45:16 2012 From: matej at laitl.cz (=?utf-8?B?TWF0xJtq?= Laitl) Date: Fri, 30 Nov 2012 01:45:16 +0100 Subject: [Cython] memoryview of extension types: warnings with gcc, errors with g++ Message-ID: <2282175.tMIWRvpovu@edgy> Hi list and Mark, it seems that C code with questionable casts is generated when using memory views of extension types. I get following warnings from gcc: extension_type_memoryview.c: In function ?__pyx_pf_25extension_type_memoryview_test_getitem?: extension_type_memoryview.c:1468:15: warning: assignment from incompatible pointer type extension_type_memoryview.c: In function ?__pyx_pf_25extension_type_memoryview_2test_getitem_typed?: extension_type_memoryview.c:1565:15: warning: assignment from incompatible pointer type extension_type_memoryview.c:1568:18: warning: assignment from incompatible pointer type And following errors if compiling in C++ mode with g++: extension_type_memoryview.c: In function ?PyObject* __pyx_pf_25extension_type_memoryview_test_getitem(PyObject*)?: extension_type_memoryview.c:1468:213: error: cannot convert ?__pyx_obj_25extension_type_memoryview_ExtensionType*? to ?PyObject*? in assignment extension_type_memoryview.c: In function ?PyObject* __pyx_pf_25extension_type_memoryview_2test_getitem_typed(PyObject*)?: extension_type_memoryview.c:1565:213: error: cannot convert ?__pyx_obj_25extension_type_memoryview_ExtensionType*? to ?PyObject*? in assignment extension_type_memoryview.c:1568:20: error: cannot convert ?PyObject*? to ?__pyx_obj_25extension_type_memoryview_ExtensionType*? in assignment I get exactly the same error when --cplus is passed to Cython and extension_type_memoryview.cpp is generated/compiled. ...which currently prevents my project combining C++ code with extension types and memory views to compile. :-( Test-case is attached. Cython 0.17.2. Regards, Mat?j -------------- next part -------------- # mode: run # tags: numpy, cpp import numpy as np cdef class ExtensionType(object): cdef public int dummy def __init__(self, n): self.dummy = n items = [ExtensionType(1), ExtensionType(2)] cdef ExtensionType[:] view = np.array(items, dtype=ExtensionType) def test_getitem(): """ >>> test_getitem() 1 2 """ for i in range(view.shape[0]): item = view[i] print item.dummy def test_getitem_typed(): """ >>> test_getitem_typed() 1 2 """ cdef ExtensionType item for i in range(view.shape[0]): item = view[i] print item.dummy From markflorisson88 at gmail.com Fri Nov 30 13:50:08 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 30 Nov 2012 12:50:08 +0000 Subject: [Cython] memoryview of extension types: warnings with gcc, errors with g++ In-Reply-To: <2282175.tMIWRvpovu@edgy> References: <2282175.tMIWRvpovu@edgy> Message-ID: On 30 November 2012 00:45, Mat?j Laitl wrote: > Hi list and Mark, > it seems that C code with questionable casts is generated when using memory > views of extension types. I get following warnings from gcc: > > extension_type_memoryview.c: In function ?__pyx_pf_25extension_type_memoryview_test_getitem?: > extension_type_memoryview.c:1468:15: warning: assignment from incompatible pointer type > extension_type_memoryview.c: In function ?__pyx_pf_25extension_type_memoryview_2test_getitem_typed?: > extension_type_memoryview.c:1565:15: warning: assignment from incompatible pointer type > extension_type_memoryview.c:1568:18: warning: assignment from incompatible pointer type > > And following errors if compiling in C++ mode with g++: > extension_type_memoryview.c: In function ?PyObject* __pyx_pf_25extension_type_memoryview_test_getitem(PyObject*)?: > extension_type_memoryview.c:1468:213: error: cannot convert ?__pyx_obj_25extension_type_memoryview_ExtensionType*? to ?PyObject*? in assignment > extension_type_memoryview.c: In function ?PyObject* __pyx_pf_25extension_type_memoryview_2test_getitem_typed(PyObject*)?: > extension_type_memoryview.c:1565:213: error: cannot convert ?__pyx_obj_25extension_type_memoryview_ExtensionType*? to ?PyObject*? in assignment > extension_type_memoryview.c:1568:20: error: cannot convert ?PyObject*? to ?__pyx_obj_25extension_type_memoryview_ExtensionType*? in assignment > > I get exactly the same error when --cplus is passed to Cython and > extension_type_memoryview.cpp is generated/compiled. > > ...which currently prevents my project combining C++ code with extension > types and memory views to compile. :-( Test-case is attached. Cython 0.17.2. > > Regards, > Mat?j Thanks for the report Mat?j, I thought we had a test for that. I'll look into it, should be an easy fix.