From stefan_ml at behnel.de Sat Sep 1 08:58:57 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 01 Sep 2012 08:58:57 +0200 Subject: [Cython] Cython 0.17 final released Message-ID: <5041B231.7020905@behnel.de> Hello everyone, on behalf of the Cython project team, I'm proud to announce the final release of Cython 0.17. This is another major step forward in the development of the language that will make life easier for a lot of users, rounds up some rough edges of the compiler and adds (preliminary) support for CPython 3.3 and PyPy. It is also the first final release of an implementation of PEP 380 (generator delegation), before it will eventually appear in CPython 3.3. Download: http://cython.org/release/Cython-0.17.tar.gz Release notes: http://wiki.cython.org/ReleaseNotes-0.17 Documentation: http://docs.cython.org/ Major features of this release include: * vastly improved integration with the C++ STL containers http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library http://docs.cython.org/src/tutorial/strings.html#c-strings * "yield from" delegation between generators (PEP 380) http://www.python.org/dev/peps/pep-0380/ * alpha quality support for PyPy (via cpyext) http://docs.cython.org/src/userguide/pypy.html Several other features and improvements are listed in the release notes: http://wiki.cython.org/ReleaseNotes-0.17 Have fun, Stefan From robertwb at gmail.com Sun Sep 2 06:05:18 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 1 Sep 2012 21:05:18 -0700 Subject: [Cython] [cython-users] Cython 0.17 final released In-Reply-To: <5041B231.7020905@behnel.de> References: <5041B231.7020905@behnel.de> Message-ID: I'm assuming cython-devel is a proper subset of cython-users, but just wanted to say hooray and thanks for all the work that wen into this release (both new development and pushing the actual release to completion). Is there a plan to announce more broadly? (E.g. python-announce, if you haven't already (moderation adds a bit of latency).) I know this was brought up a couple of months ago, but we seem to be falling into a pattern of thinking about doing a bugfix release, then saying a full release is around the corner anyways, and then not actually getting it out for quite a while. I propose we start an actual bugfix branch from which we can cut minor releases at (nearly) any time, and certainly with much less hassle. Thoughts? - Robert On Fri, Aug 31, 2012 at 11:58 PM, Stefan Behnel wrote: > Hello everyone, > > on behalf of the Cython project team, I'm proud to announce the final > release of Cython 0.17. > > This is another major step forward in the development of the language that > will make life easier for a lot of users, rounds up some rough edges of the > compiler and adds (preliminary) support for CPython 3.3 and PyPy. It is > also the first final release of an implementation of PEP 380 (generator > delegation), before it will eventually appear in CPython 3.3. > > > Download: http://cython.org/release/Cython-0.17.tar.gz > > Release notes: http://wiki.cython.org/ReleaseNotes-0.17 > > Documentation: http://docs.cython.org/ > > > Major features of this release include: > > * vastly improved integration with the C++ STL containers > > http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library > > http://docs.cython.org/src/tutorial/strings.html#c-strings > > * "yield from" delegation between generators (PEP 380) > > http://www.python.org/dev/peps/pep-0380/ > > * alpha quality support for PyPy (via cpyext) > > http://docs.cython.org/src/userguide/pypy.html > > > Several other features and improvements are listed in the release notes: > > http://wiki.cython.org/ReleaseNotes-0.17 > > > Have fun, > > Stefan From stefan_ml at behnel.de Sun Sep 2 08:49:25 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Sep 2012 08:49:25 +0200 Subject: [Cython] Cython 0.17 final released In-Reply-To: References: <5041B231.7020905@behnel.de> Message-ID: <50430175.1020104@behnel.de> Robert Bradshaw, 02.09.2012 06:05: > Is there a plan to announce more broadly? (E.g. python-announce, if > you haven't already (moderation adds a bit of latency).) I had already sent it to the python-announce list and also made an announcement on c.l.py now. :) > I know this was brought up a couple of months ago, but we seem to be > falling into a pattern of thinking about doing a bugfix release, then > saying a full release is around the corner anyways, and then not > actually getting it out for quite a while. I propose we start an > actual bugfix branch from which we can cut minor releases at (nearly) > any time, and certainly with much less hassle. Thoughts? +1, I'll push the master into the release branch and update the master version to 0.18a0. Regarding 0.18, I won't be able to review any of Mark's changes in his array expressions branch, but I take it that Dag did, so I assume that they should go in right next (after a necessary rebase and maybe some adaptations for the external minivect) and then get consolidated there (if necessary). Looks like they'll make the first big feature for the next release (which in turn makes Mark the perfect "volunteer" for the next release manager ;) ). Stefan From robertwb at gmail.com Sun Sep 2 09:17:17 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sun, 2 Sep 2012 00:17:17 -0700 Subject: [Cython] Cython 0.17 final released In-Reply-To: <50430175.1020104@behnel.de> References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> Message-ID: On Sat, Sep 1, 2012 at 11:49 PM, Stefan Behnel wrote: > Robert Bradshaw, 02.09.2012 06:05: >> Is there a plan to announce more broadly? (E.g. python-announce, if >> you haven't already (moderation adds a bit of latency).) > > I had already sent it to the python-announce list and also made an > announcement on c.l.py now. :) > > >> I know this was brought up a couple of months ago, but we seem to be >> falling into a pattern of thinking about doing a bugfix release, then >> saying a full release is around the corner anyways, and then not >> actually getting it out for quite a while. I propose we start an >> actual bugfix branch from which we can cut minor releases at (nearly) >> any time, and certainly with much less hassle. Thoughts? > > +1, I'll push the master into the release branch and update the master > version to 0.18a0. Perhaps we should use the -pre or -post or -dev marker, as it's not really a marked alpha per say? What about making a 0.17 branch, on which we can do 0.17.1, etc. if necessary. I don't think we're to the point of needing to backport our bugfixes to previous releases, but that will make things clearer. When the 0.18 series is ready, we can make a new branch and work on the release from there (possibly merging in master frequently, but not as a requirement). Alternatively name it bugfix and let it be regularly merged into main, but we can cut minor releases from it. (Release is a bit ambiguous as to its relationship to the other branches). > Regarding 0.18, I won't be able to review any of Mark's changes in his > array expressions branch, but I take it that Dag did, so I assume that they > should go in right next (after a necessary rebase and maybe some > adaptations for the external minivect) and then get consolidated there (if > necessary). Looks like they'll make the first big feature for the next > release (which in turn makes Mark the perfect "volunteer" for the next > release manager ;) ). I'd be up for that :). - Robert From stefan_ml at behnel.de Sun Sep 2 09:40:23 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Sep 2012 09:40:23 +0200 Subject: [Cython] Cython 0.17 final released In-Reply-To: References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> Message-ID: <50430D67.5050608@behnel.de> Robert Bradshaw, 02.09.2012 09:17: > On Sat, Sep 1, 2012 at 11:49 PM, Stefan Behnel wrote: >> Robert Bradshaw, 02.09.2012 06:05: >>> I know this was brought up a couple of months ago, but we seem to be >>> falling into a pattern of thinking about doing a bugfix release, then >>> saying a full release is around the corner anyways, and then not >>> actually getting it out for quite a while. I propose we start an >>> actual bugfix branch from which we can cut minor releases at (nearly) >>> any time, and certainly with much less hassle. Thoughts? >> >> +1, I'll push the master into the release branch and update the master >> version to 0.18a0. > > Perhaps we should use the -pre or -post or -dev marker, as it's not > really a marked alpha per say? Right, it's not going to be used in a release, so we don't need it to be parsable. "0.18-pre" it is then. > What about making a 0.17 branch, on which we can do 0.17.1, etc. if > necessary. I don't think we're to the point of needing to backport our > bugfixes to previous releases, but that will make things clearer. When > the 0.18 series is ready, we can make a new branch and work on the > release from there (possibly merging in master frequently, but not as > a requirement). Alternatively name it bugfix and let it be regularly > merged into main, but we can cut minor releases from it. (Release is a > bit ambiguous as to its relationship to the other branches). Ok, "0.17" then. Stefan From robertwb at gmail.com Wed Sep 5 02:51:33 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 4 Sep 2012 17:51:33 -0700 Subject: [Cython] Non-profit 501(c)(1) Cython foundation Message-ID: NumFOCUS has graciously offered to help Cython get non-profit US tax status as part of their application as a part of a "Group Exception." Basically, this gives us all the benefits of a non-profit (including being able to receive and spend tax-favored donations) without having to go through the enormous amount of paperwork involved, both upfront (to apply) and ongoing. For more context, see https://groups.google.com/forum/?fromgroups=#!topic/numfocus/moVBdZDLqmI Though we don't have any immediate need of such a status, doing it now will be much easier than trying to do it later (especially if there's a tight timeframe when we recognize the need). I went ahead and applied for an EIN for a "Cython Foundation" and they have some paperwork to sign (see below). Any thoughts/concerns/suggestions? - Robert ---------- Forwarded message ---------- From: Leah Holdridge Date: Tue, Sep 4, 2012 at 3:32 PM Subject: Group Exemption To: Ond?ej ?ert?k , Brian Granger , Robert Bradshaw Cc: Numfocus admin Hi Ondrej, Brian, and Robert, The good news is; the only items I need from you for the Group Exemption are the 2 documents I've attached needing your signature. The first is the Articles of Association. This is the uniform governing instrument that states your nonprofit tax-exempt purpose in 501(c)(3) language. It also states that you are a subordinate organization under general supervision of NumFOCUS. This is regarding the responsibility NumFOCUS has in ensuring that all it's subordinates continue to qualify for tax-exempt status. I need 2 signatures, it should be officers, but since you are not a formal corporation at this time, it can really be anyone involved. It would be best if they are US citizens. The other document is the letter from you giving written authorization to NumFOCUS to be a central organization. I have listed each of you as president of your group, if your have another title, or don't have one at all, that's no problem. Also, it states in the letter that as a board you have voted on and approved this. Since you have no formal governing body or bylaws, this can be interpreted very loosely, an email or discussion is fine. Ondrej, I didn't have your address, so that needs to be put in SymPy's letter. Brian, I need your EIN, the actual name you registered with the IRS when you applied for your EIN ( I used IPython on the attached documents, but wasn't sure if that needs to be corrected), your mailing address, and brief description with expenditures. Now for the not so good news; I need these back tomorrow or first thing Thursday morning so I can send off the application Thursday. You can print the documents off, sign, and fax back to me at: 800-753-8308. Thanks. Have a good one, Leah -------------- next part -------------- A non-text attachment was scrubbed... Name: Cython ARTICLES OF ASSOCIATION .docx Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document Size: 136661 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Cython Request letter.doc Type: application/msword Size: 24576 bytes Desc: not available URL: From robertwb at gmail.com Wed Sep 5 07:15:00 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 4 Sep 2012 22:15:00 -0700 Subject: [Cython] Non-profit 501(c)(1) Cython foundation In-Reply-To: References: Message-ID: Sorry, that's 501(c)(3) (charitable org). On Tue, Sep 4, 2012 at 5:51 PM, Robert Bradshaw wrote: > NumFOCUS has graciously offered to help Cython get non-profit US tax > status as part of their application as a part of a "Group Exception." > Basically, this gives us all the benefits of a non-profit (including > being able to receive and spend tax-favored donations) without having > to go through the enormous amount of paperwork involved, both upfront > (to apply) and ongoing. For more context, see > > https://groups.google.com/forum/?fromgroups=#!topic/numfocus/moVBdZDLqmI > > Though we don't have any immediate need of such a status, doing it now > will be much easier than trying to do it later (especially if there's > a tight timeframe when we recognize the need). I went ahead and > applied for an EIN for a "Cython Foundation" and they have some > paperwork to sign (see below). Any thoughts/concerns/suggestions? > > - Robert > > > ---------- Forwarded message ---------- > From: Leah Holdridge > Date: Tue, Sep 4, 2012 at 3:32 PM > Subject: Group Exemption > To: Ond?ej ?ert?k , Brian Granger > , Robert Bradshaw > Cc: Numfocus admin > > > > Hi Ondrej, Brian, and Robert, > > > The good news is; the only items I need from you for the Group > Exemption are the 2 documents I've attached needing your signature. > > The first is the Articles of Association. This is the uniform > governing instrument that states your nonprofit tax-exempt purpose in > 501(c)(3) language. It also states that you are a subordinate > organization under general supervision of NumFOCUS. This is regarding > the responsibility NumFOCUS has in ensuring that all it's subordinates > continue to qualify for tax-exempt status. I need 2 signatures, it > should be officers, but since you are not a formal corporation at this > time, it can really be anyone involved. It would be best if they are > US citizens. > > The other document is the letter from you giving written authorization > to NumFOCUS to be a central organization. I have listed each of you > as president of your group, if your have another title, or don't have > one at all, that's no problem. Also, it states in the letter that as > a board you have voted on and approved this. Since you have no formal > governing body or bylaws, this can be interpreted very loosely, an > email or discussion is fine. > > Ondrej, I didn't have your address, so that needs to be put in SymPy's > letter. Brian, I need your EIN, the actual name you registered with > the IRS when you applied for your EIN ( I used IPython on the attached > documents, but wasn't sure if that needs to be corrected), your > mailing address, and brief description with expenditures. > > Now for the not so good news; I need these back tomorrow or first > thing Thursday morning so I can send off the application Thursday. You > can print the documents off, sign, and fax back to me at: > 800-753-8308. > > Thanks. > > > > Have a good one, > > Leah From d.s.seljebotn at astro.uio.no Wed Sep 5 10:04:32 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 Sep 2012 10:04:32 +0200 Subject: [Cython] Non-profit 501(c)(1) Cython foundation In-Reply-To: References: Message-ID: <50470790.4080103@astro.uio.no> On 09/05/2012 02:51 AM, Robert Bradshaw wrote: > NumFOCUS has graciously offered to help Cython get non-profit US tax > status as part of their application as a part of a "Group Exception." > Basically, this gives us all the benefits of a non-profit (including > being able to receive and spend tax-favored donations) without having > to go through the enormous amount of paperwork involved, both upfront > (to apply) and ongoing. For more context, see > > https://groups.google.com/forum/?fromgroups=#!topic/numfocus/moVBdZDLqmI > > Though we don't have any immediate need of such a status, doing it now > will be much easier than trying to do it later (especially if there's > a tight timeframe when we recognize the need). I went ahead and > applied for an EIN for a "Cython Foundation" and they have some > paperwork to sign (see below). Any thoughts/concerns/suggestions? Nice. If it is preferred to have another American sign, I guess make William Stein a member of the board and get his signature? Otherwise I think Stefan should give the other signature. Dag > > - Robert > > > ---------- Forwarded message ---------- > From: Leah Holdridge > Date: Tue, Sep 4, 2012 at 3:32 PM > Subject: Group Exemption > To: Ond?ej ?ert?k , Brian Granger > , Robert Bradshaw > Cc: Numfocus admin > > > > Hi Ondrej, Brian, and Robert, > > > The good news is; the only items I need from you for the Group > Exemption are the 2 documents I've attached needing your signature. > > The first is the Articles of Association. This is the uniform > governing instrument that states your nonprofit tax-exempt purpose in > 501(c)(3) language. It also states that you are a subordinate > organization under general supervision of NumFOCUS. This is regarding > the responsibility NumFOCUS has in ensuring that all it's subordinates > continue to qualify for tax-exempt status. I need 2 signatures, it > should be officers, but since you are not a formal corporation at this > time, it can really be anyone involved. It would be best if they are > US citizens. > > The other document is the letter from you giving written authorization > to NumFOCUS to be a central organization. I have listed each of you > as president of your group, if your have another title, or don't have > one at all, that's no problem. Also, it states in the letter that as > a board you have voted on and approved this. Since you have no formal > governing body or bylaws, this can be interpreted very loosely, an > email or discussion is fine. > > Ondrej, I didn't have your address, so that needs to be put in SymPy's > letter. Brian, I need your EIN, the actual name you registered with > the IRS when you applied for your EIN ( I used IPython on the attached > documents, but wasn't sure if that needs to be corrected), your > mailing address, and brief description with expenditures. > > Now for the not so good news; I need these back tomorrow or first > thing Thursday morning so I can send off the application Thursday. You > can print the documents off, sign, and fax back to me at: > 800-753-8308. > > Thanks. > > > > Have a good one, > > Leah > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From njs at pobox.com Sat Sep 8 22:14:47 2012 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 8 Sep 2012 21:14:47 +0100 Subject: [Cython] Fwd: [numpy] MAINT: silence Cython warnings about changes dtype/ufunc size. (#432) In-Reply-To: References: Message-ID: Hi Cythoneers, Ralf just proposed this pull request for numpy, to unconditionally install a warning filter to silence all "numpy.dtype size changed", "numpy.ufunc size changed" warnings that Cython likes to spit out. See the links below for details. Figured you should have a chance to comment, anyway. -n ---------- Forwarded message ---------- From: Ralf Gommers Date: Sat, Sep 8, 2012 at 9:00 PM Subject: [numpy] MAINT: silence Cython warnings about changes dtype/ufunc size. (#432) To: numpy/numpy These warnings are visible whenever you import scipy (or another package) that was compiled against an older numpy than is installed. For example compiled against 1.5.1, like current scipy binaries are, and used with 1.7.0. These warnings aren't useful; if numpy would really break its ABI it would be noticed in no time without these warnings. ------------------------------ You can merge this Pull Request by running: git pull https://github.com/rgommers/numpy silence-cython-warnings Or view, comment on, or merge it at: https://github.com/numpy/numpy/pull/432 Commit Summary - MAINT: silence Cython warnings about changes dtype/ufunc size. File Changes - *M* numpy/__init__.py (6) Patch Links - https://github.com/numpy/numpy/pull/432.patch - https://github.com/numpy/numpy/pull/432.diff ? Reply to this email directly or view it on GitHub. -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Sat Sep 8 23:01:48 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Sat, 08 Sep 2012 23:01:48 +0200 Subject: [Cython] Fwd: [numpy] MAINT: silence Cython warnings about changes dtype/ufunc size. (#432) In-Reply-To: References: Message-ID: <72c3eca4-5516-45f2-8aca-9713f8f73f40@email.android.com> Nathaniel Smith wrote: >Hi Cythoneers, > >Ralf just proposed this pull request for numpy, to unconditionally >install >a warning filter to silence all "numpy.dtype size changed", >"numpy.ufunc >size changed" warnings that Cython likes to spit out. See the links >below >for details. Figured you should have a chance to comment, anyway. I'm +1 on this, I think it is great. DS > >-n > >---------- Forwarded message ---------- >From: Ralf Gommers >Date: Sat, Sep 8, 2012 at 9:00 PM >Subject: [numpy] MAINT: silence Cython warnings about changes >dtype/ufunc >size. (#432) >To: numpy/numpy > > >These warnings are visible whenever you import scipy (or another >package) >that >was compiled against an older numpy than is installed. For example >compiled >against 1.5.1, like current scipy binaries are, and used with 1.7.0. > >These warnings aren't useful; if numpy would really break its ABI it >would >be >noticed in no time without these warnings. >------------------------------ >You can merge this Pull Request by running: > > git pull https://github.com/rgommers/numpy silence-cython-warnings > >Or view, comment on, or merge it at: > > https://github.com/numpy/numpy/pull/432 >Commit Summary > > - MAINT: silence Cython warnings about changes dtype/ufunc size. > >File Changes > > - *M* numpy/__init__.py (6) > >Patch Links > > - https://github.com/numpy/numpy/pull/432.patch > - https://github.com/numpy/numpy/pull/432.diff > > ? >Reply to this email directly or view it on >GitHub. >_______________________________________________ >cython-devel mailing list >cython-devel at python.org >http://mail.python.org/mailman/listinfo/cython-devel -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. From robertwb at gmail.com Sun Sep 9 07:48:48 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 8 Sep 2012 22:48:48 -0700 Subject: [Cython] Fwd: [numpy] MAINT: silence Cython warnings about changes dtype/ufunc size. (#432) In-Reply-To: <72c3eca4-5516-45f2-8aca-9713f8f73f40@email.android.com> References: <72c3eca4-5516-45f2-8aca-9713f8f73f40@email.android.com> Message-ID: On Sat, Sep 8, 2012 at 2:01 PM, Dag Sverre Seljebotn wrote: > > > Nathaniel Smith wrote: > >>Hi Cythoneers, >> >>Ralf just proposed this pull request for numpy, to unconditionally >>install >>a warning filter to silence all "numpy.dtype size changed", >>"numpy.ufunc >>size changed" warnings that Cython likes to spit out. See the links >>below >>for details. Figured you should have a chance to comment, anyway. > > I'm +1 on this, I think it is great. I think it makes sense for NumPy, where the extension is written by hand and great care is taken to maintain ABI compatibility, but the warning is too useful to turn off in general for all of Cython. - Robert From byrnes at wildpumpkin.net Mon Sep 10 22:51:23 2012 From: byrnes at wildpumpkin.net (Robert Byrnes) Date: Mon, 10 Sep 2012 16:51:23 -0400 Subject: [Cython] Cython-0.17 builtin type import bug Message-ID: <201209102051.q8AKpNLN016443@whimsy.wildpumpkin.net> If a builtin type that differs between Python 2 and 3 (e.g., unicode) is imported, then malformed C code is emitted by Cython-0.17: there is an extraneous, unbalanced #if. Here's a small reproducer: shell$ cat foo.pyx cdef extern from "Python.h": ctypedef class __builtin__.unicode [object PyUnicodeObject]: pass shell$ cython foo.pyx warning: foo.pyx:2:4: unicode already a builtin Cython type shell$ gcc -c -fPIC -I/usr/include/python2.6 foo.c foo.c:5:1: error: unterminated #else foo.c: In function 'initfoo': foo.c:574: error: expected declaration or statement at end of input foo.c:535: error: label '__pyx_L1_error' used but not defined This works for Cython-0.16 and earlier. I think this was broken by ... https://github.com/cython/cython/commit/558f4ef1c48ce091fa15b9319eb0f92cd4758f09 This patch for ModuleNode.generate_type_import_call seems to fix it: --- a/Cython/Compiler/ModuleNode.py +++ b/Cython/Compiler/ModuleNode.py @@ -2281,7 +2281,6 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode): module_name = '__Pyx_BUILTIN_MODULE_NAME' if type.name in Code.non_portable_builtins_map: condition, replacement = Code.non_portable_builtins_map[type.name] - code.putln("#if %s" % condition) if objstruct in Code.basicsize_builtins_map: # Some builtin types have a tp_basicsize which differs from sizeof(...): sizeof_objstruct = Code.basicsize_builtins_map[objstruct] Thanks, -- Bob From wesmckinn at gmail.com Tue Sep 11 00:21:02 2012 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 10 Sep 2012 18:21:02 -0400 Subject: [Cython] Hitting gcc 4.6 bug with Cython 0.17 and pandas Message-ID: This issue came up earlier today. Everything works fine for me with gcc 4.6.1 and Cython 0.17, but maybe there is a bug being hit in gcc 4.6. Any debugging help would be appreciated (since this affects EC2 AMIs...) http://github.com/pydata/pandas/issues/1880 thanks, Wes From brad.froehle at gmail.com Tue Sep 11 00:34:00 2012 From: brad.froehle at gmail.com (Bradley M. Froehle) Date: Mon, 10 Sep 2012 15:34:00 -0700 Subject: [Cython] Cython 0.17 final released In-Reply-To: <50430D67.5050608@behnel.de> References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> <50430D67.5050608@behnel.de> Message-ID: Any chance we can tag the official 0.17 release in Github as well? -Brad On Sun, Sep 2, 2012 at 12:40 AM, Stefan Behnel wrote: > Robert Bradshaw, 02.09.2012 09:17: >> On Sat, Sep 1, 2012 at 11:49 PM, Stefan Behnel wrote: >>> Robert Bradshaw, 02.09.2012 06:05: >>>> I know this was brought up a couple of months ago, but we seem to be >>>> falling into a pattern of thinking about doing a bugfix release, then >>>> saying a full release is around the corner anyways, and then not >>>> actually getting it out for quite a while. I propose we start an >>>> actual bugfix branch from which we can cut minor releases at (nearly) >>>> any time, and certainly with much less hassle. Thoughts? >>> >>> +1, I'll push the master into the release branch and update the master >>> version to 0.18a0. >> >> Perhaps we should use the -pre or -post or -dev marker, as it's not >> really a marked alpha per say? > > Right, it's not going to be used in a release, so we don't need it to be > parsable. "0.18-pre" it is then. > > >> What about making a 0.17 branch, on which we can do 0.17.1, etc. if >> necessary. I don't think we're to the point of needing to backport our >> bugfixes to previous releases, but that will make things clearer. When >> the 0.18 series is ready, we can make a new branch and work on the >> release from there (possibly merging in master frequently, but not as >> a requirement). Alternatively name it bugfix and let it be regularly >> merged into main, but we can cut minor releases from it. (Release is a >> bit ambiguous as to its relationship to the other branches). > > Ok, "0.17" then. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From fperez.net at gmail.com Tue Sep 11 02:41:01 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Mon, 10 Sep 2012 17:41:01 -0700 Subject: [Cython] Cython 0.17 final released In-Reply-To: References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> Message-ID: First, congratulations and thanks for keeping such an important project moving forward!! On Sun, Sep 2, 2012 at 12:17 AM, Robert Bradshaw wrote: > > What about making a 0.17 branch, on which we can do 0.17.1, etc. if > necessary. I don't think we're to the point of needing to backport our > bugfixes to previous releases, but that will make things clearer. When > the 0.18 series is ready, we can make a new branch and work on the > release from there (possibly merging in master frequently, but not as > a requirement). Alternatively name it bugfix and let it be regularly > merged into main, but we can cut minor releases from it. (Release is a > bit ambiguous as to its relationship to the other branches). In case it's of any use to you guys, feel free to grab our tools for that; we follow precisely that pattern and we're just about to cut a backports-only 0.13.1 release soon: https://github.com/ipython/ipython/blob/master/tools/backport_pr.py Min wrote this tool and from what he tells me, it has really minimized the overhead of keeping a clean backports-only branch available for release. Cheers, f From robertwb at gmail.com Tue Sep 11 09:21:10 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 11 Sep 2012 00:21:10 -0700 Subject: [Cython] Cython 0.17 final released In-Reply-To: References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> Message-ID: On Mon, Sep 10, 2012 at 5:41 PM, Fernando Perez wrote: > First, congratulations and thanks for keeping such an important > project moving forward!! > > On Sun, Sep 2, 2012 at 12:17 AM, Robert Bradshaw wrote: >> >> What about making a 0.17 branch, on which we can do 0.17.1, etc. if >> necessary. I don't think we're to the point of needing to backport our >> bugfixes to previous releases, but that will make things clearer. When >> the 0.18 series is ready, we can make a new branch and work on the >> release from there (possibly merging in master frequently, but not as >> a requirement). Alternatively name it bugfix and let it be regularly >> merged into main, but we can cut minor releases from it. (Release is a >> bit ambiguous as to its relationship to the other branches). > > In case it's of any use to you guys, feel free to grab our tools for > that; we follow precisely that pattern and we're just about to cut a > backports-only 0.13.1 release soon: > > https://github.com/ipython/ipython/blob/master/tools/backport_pr.py > > Min wrote this tool and from what he tells me, it has really minimized > the overhead of keeping a clean backports-only branch available for > release. Thanks. Just to clarify your process is to merge the fix into the main branch, and then backport it by creating a new commit (with the above script) for the old releases? - Robert From barcc at gmx.de Wed Sep 12 12:49:57 2012 From: barcc at gmx.de (B. Clausius) Date: Wed, 12 Sep 2012 12:49:57 +0200 Subject: [Cython] cython --create-listing fails Message-ID: <505068D5.1020000@gmx.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, when running: $ cython --create-listing -o build/temp.linux-x86_64-2.7/pybiklib/gldraw_c.c build/temp.linux-x86_64-2.7/pybiklib/gldraw_c.pyx I get the error: Traceback (most recent call last): File "/usr/bin/cython", line 8, in main(command_line = 1) File "/usr/lib/pymodules/python2.7/Cython/Compiler/Main.py", line 662, in main result = compile(sources, options) File "/usr/lib/pymodules/python2.7/Cython/Compiler/Main.py", line 637, in compile return compile_multiple(source, options) File "/usr/lib/pymodules/python2.7/Cython/Compiler/Main.py", line 609, in compile_multiple result = run_pipeline(source, options) File "/usr/lib/pymodules/python2.7/Cython/Compiler/Main.py", line 472, in run_pipeline context.setup_errors(options, result) File "/usr/lib/pymodules/python2.7/Cython/Compiler/Main.py", line 399, in setup_errors result.listing_file = Utils.replace_suffix(source, ".lis") NameError: global name 'source' is not defined I'm using Cython 0.15.1 . The patch is against git as it seems the code there is same but tested only for 0.15.1. B.C. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQUGjJAAoJEEDg0A3D0GN4Yj0QAKYHfo1C7bwl8a2pZi+DPrrE N0T5KAzu3RjYgacw0MpC5hb+L1X6J5Yi3BAdIp4ertCY6xeKD6+lQt2q+zHK0De3 0CI4ivnh+yH/x5hvjRjDprrpotKc/tre0BWMpKpchw2ohZ21PRDmxpOsjFzTmyC0 V8TO0AG0hzpabzI9qNZRwyBz/GOzfFwyumKBDOv/z/2n2LDDBO+k10orfXrfhjdI uFE6m1YAA4vK0apTi0F6GokJNq+vqT2pRNEzO/Fsek7kRxoNCAUrbzL0aKMj8wsz M4wIUq8TE3R2d5cvEt+ibQK0+7OydoKoaiCd3BfumVsj5SBRio/GcMetJA1lThUn W8e1yPtzPO/GlSatdvRiIJFzyzlX1ZVAfFXD419Gq0/X23/GK+j9k1rHQzg8e3+j 7yVph3DLfZfnQqmBpD3Xl4NCp7zYq7QgfdsB1GNzub/PEatfcv2DVLlnCDCTKgGu j5/r/0dwrOkb1q3SGtONo/dAovSi0oIbW3Wfe4ebB8S1HVo9pFpPDIqc1g3xXRA2 laqc7vBAcE5ZfS5/uDmW74PMV1khDeDkhmhoEGSQbFTSSC40DwufvyBAPMyzjjvf 9JAvOp5FS9RS0020KPKIlOtxQXA+CrJSIbaQZu6m/nmH17a60QnW8kwZ799/Pqgd IsacvSJvpYKHtQbg9JCJ =Hav4 -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: create-listing.patch Type: text/x-patch Size: 579 bytes Desc: not available URL: From barcc at gmx.de Wed Sep 12 14:36:05 2012 From: barcc at gmx.de (B. Clausius) Date: Wed, 12 Sep 2012 14:36:05 +0200 Subject: [Cython] usage does not match the options Message-ID: <505081B5.8070509@gmx.de> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I found that the options shown by "cython -h" are different to the options parsed by cython. Usage: --warning-error Parser: --warning-errors The parser does not recognize "-q" and "-quiet" (Patch attached) There are other options that are not in the usage: -+ -z, --pre-import --convert-range --no-c-in-traceback --disable-function-redefinition --old-style-globals --debug -h, --help B.C. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQIcBAEBAgAGBQJQUIG1AAoJEEDg0A3D0GN4lN0QAKicvLAAcVqvJ6+12ErOSLxx hzvJcIvjR9EBys3QMzq6+wfGY8JdzWA4ofn6VI+xtLkFDt9NhlpvB64UN1TUy1Nm zj5IopRVV9VuKAvWxHoxk7QsuR2PmOd3ZR7d+v4xOxncfKmp1NNYYwuSvF9o232M gdABJ3U34hGXhWTtVt21WkFqBgIQFnuWsCCzOokNL/rYuUEu1/AalFgy6Eko8xs8 V55bGXAFEgNCGJb/DPWglsTqoS3XJO4Wr7nt/N7ca6ejHo6Q0acA8GEcBikXM7kA KI7swe/FwcNzz3FR0JflhbnzVltrR/bVGuhtjdElH2CNfSeX0lLIeihl9qmANYiZ oGnl5n2Wmx/jUaFiEVmXa1ry+0xaNU10n7lv9zfAJYHt4xzF0HOmyudmviU34SVi 2w+SVPdBNH/08R0C3JPkcrLxN3L9RkXWzv6+bEsdtTGrWycQyXSLpVJn66OtAeKl LoBJw2wzfibazKIWO7PuNW4VTz5Mcn4m8NG/ujEpw3WUE97mqRZRm9cp3opaeIjh uZeWLUsEgn2GEMAwVrg10ILdverD08RX7ovyUqKCP84W03MV5fBHQISNAW2rS1ig KXf1zm0lYr+W1MN9BnGQtqUTFSzxpnNJZ4KhDio+kAMc4bqbSk6R2ste2XXbS6/j JGCc02zjM7TtSuw33oeJ =8zfk -----END PGP SIGNATURE----- -------------- next part -------------- A non-text attachment was scrubbed... Name: usage_does_not_match_the_options.patch Type: text/x-patch Size: 1289 bytes Desc: not available URL: From robertwb at gmail.com Wed Sep 12 14:54:40 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 12 Sep 2012 05:54:40 -0700 Subject: [Cython] usage does not match the options In-Reply-To: <505081B5.8070509@gmx.de> References: <505081B5.8070509@gmx.de> Message-ID: On Wed, Sep 12, 2012 at 5:36 AM, B. Clausius wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, > > I found that the options shown by "cython -h" are different to the > options parsed by cython. > > Usage: --warning-error > Parser: --warning-errors > The parser does not recognize "-q" and "-quiet" > (Patch attached) Thanks. Pushed. https://github.com/cython/cython/commit/eae1ada81dd15528b26f3d802a31f04cdd4a22f6 (Generally we prefer pull requests, but we're not that picky when people are helping out...) > There are other options that are not in the usage: > -+ > -z, --pre-import > --convert-range > --no-c-in-traceback > --disable-function-redefinition > --old-style-globals > --debug > -h, --help Yeah, none of these should be advertised. - Robert From robertwb at gmail.com Fri Sep 14 22:19:58 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 14 Sep 2012 13:19:58 -0700 Subject: [Cython] Fwd: [cython-users] can pointers be stored in Python's dict or list? In-Reply-To: References: <20120914161147.58f58ab8@dino> <20120914205458.09da1b0e@dino> Message-ID: Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases. Note that object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences). - Robert ---------- Forwarded message ---------- From: Robert Bradshaw Date: Fri, Sep 14, 2012 at 1:11 PM Subject: Re: [cython-users] can pointers be stored in Python's dict or list? To: Mark Summerfield Cc: cython-users at googlegroups.com On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield wrote: > Hi Robert, > > On Fri, 14 Sep 2012 10:45:06 -0700 > Robert Bradshaw wrote: >> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield >> wrote: >> > Hi, >> > >> > (1) >> > >> > I am using Cython to provide a Python wrapper to a C library. >> > The C library has a function which gives a handle (a pointer) which is >> > then used to call other functions. >> > >> > I want to be able to store a dictionary of name-pointer pairs so that I >> > can resuse handles. >> > >> > This doesn't seem to work with dict or list because of the >> > Cannot convert 'Thing*' to Python object >> > error. >> > >> > Right now I have a static array and a dictionary: >> > >> > cdef Thing* pointers[30] >> > indexForThingName = {} >> > >> > but I'd really like to be able to store an arbitrary number of >> > pointers, ideally in a dict. >> > >> > Is this possible? >> >> Sounds like you want >> http://docs.python.org/release/3.1.5/c-api/capsule.html > > But that means I have to drop down to C. I want to stay in Python and > Cython. You're already "in C" if you're dealing with pointers. > I was hoping that Cython offered some kind of wrapper type, > something that at the Python level would look like: > > class Pointer(object): > def __init__(self, void *pointer): > cdef void *self.pointer = pointer # Doesn't work > > that could be put in a dict or list. Then I could do things like this: > > Pointer p = c_func_that_returns_a_pointer() > mylist.append(p) > ... > Foo *q = mylist.pop() How about (untested) from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr cdef void* p = c_func_that_returns_a_pointer() mylist.append(PyCObject_FromVoidPtr(p, NULL) ... Foo *q = PyCObject_AsVoidPtr(mylist.pop()) This should work for Python 2, use capsules (at the provided link) for Python 3+. For a portable solution, you could cast the void* to a intptr_t (or long long, if that's sufficient) which converts back and forth from Python just fine (but is more hackish). > Oh well, I'll see if I can use libcpp's map... > >> > (2) >> > >> > Also, I tried doing >> > >> > from libc.stdio cimport printf >> > >> > and then in my code: >> > >> > printf("%p", pointer) >> > >> > but although it compiled without error, nothing was printed. >> >> Did it perhaps not flush? You also might want to try adding a newline. > > I changed it to printf("%p\n", pointer) and now I get > Cannot convert 'void *' to Python object Are you still cimporting printf? - Robert From d.s.seljebotn at astro.uio.no Fri Sep 14 23:29:59 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 14 Sep 2012 23:29:59 +0200 Subject: [Cython] =?utf-8?q?Fwd=3A_=5Bcython-users=5D_can_pointers_be_stor?= =?utf-8?q?ed_in_Python=27s=09dict_or_list=3F?= In-Reply-To: References: <20120914161147.58f58ab8@dino> <20120914205458.09da1b0e@dino> Message-ID: Isn't there a case for converting to/from ctypes pointers rather than capsules? And if capsules, what would the secret word be? Hmm... (sorry for the top post) DS -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Robert Bradshaw wrote: Given the CObject vs Capsule differences in Py2 vs Py3, and the clumsiness in using them (see below), any thoughts on automatically converting void* to/from a Python object? There is the sticky issue of pointer lifetime management, but we could assume the lifetime is managed entirely in C in most cases. Note that object already has a specific meaning that we can't hijack, but perhaps non-void* would be fine to do implicitly? Also, if we built this into the language, could we provide type safety via the name/description attribute? Something like this could be really handy to have and isn't as easily done as a library (especially given the 2/3 differences). - Robert ---------- Forwarded message ---------- From: Robert Bradshaw Date: Fri, Sep 14, 2012 at 1:11 PM Subject: Re: [cython-users] can pointers be stored in Python's dict or list? To: Mark Summerfield Cc: cython-users at googlegroups.com On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield wrote: > Hi Robert, > > On Fri, 14 Sep 2012 10:45:06 -0700 > Robert Bradshaw wrote: >> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield >> wrote: >> > Hi, >> > >> > (1) >> > >> > I am using Cython to provide a Python wrapper to a C library. >> > The C library has a function which gives a handle (a pointer) which is >> > then used to call other functions. >> > >> > I want to be able to store a dictionary of name-pointer pairs so that I >> > can resuse handles. >> > >> > This doesn't seem to work with dict or list because of the >> > Cannot convert 'Thing*' to Python object >> > error. >> > >> > Right now I have a static array and a dictionary: >> > >> > cdef Thing* pointers[30] >> > indexForThingName = {} >> > >> > but I'd really like to be able to store an arbitrary number of >> > pointers, ideally in a dict. >> > >> > Is this possible? >> >> Sounds like you want >> http://docs.python.org/release/3.1.5/c-api/capsule.html > > But that means I have to drop down to C. I want to stay in Python and > Cython. You're already "in C" if you're dealing with pointers. > I was hoping that Cython offered some kind of wrapper type, > something that at the Python level would look like: > > class Pointer(object): > def __init__(self, void *pointer): > cdef void *self.pointer = pointer # Doesn't work > > that could be put in a dict or list. Then I could do things like this: > > Pointer p = c_func_that_returns_a_pointer() > mylist.append(p) > ... > Foo *q = mylist.pop() How about (untested) from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr cdef void* p = c_func_that_returns_a_pointer() mylist.append(PyCObject_FromVoidPtr(p, NULL) ... Foo *q = PyCObject_AsVoidPtr(mylist.pop()) This should work for Python 2, use capsules (at the provided link) for Python 3+. For a portable solution, you could cast the void* to a intptr_t (or long long, if that's sufficient) which converts back and forth from Python just fine (but is more hackish). > Oh well, I'll see if I can use libcpp's map... > >> > (2) >> > >> > Also, I tried doing >> > >> > from libc.stdio cimport printf >> > >> > and then in my code: >> > >> > printf("%p", pointer) >> > >> > but although it compiled without error, nothing was printed. >> >> Did it perhaps not flush? You also might want to try adding a newline. > > I changed it to printf("%p\n", pointer) and now I get > Cannot convert 'void *' to Python object Are you still cimporting printf? - Robert _____________________________________________ cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Sat Sep 15 00:39:56 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 14 Sep 2012 15:39:56 -0700 Subject: [Cython] Fwd: [cython-users] can pointers be stored in Python's dict or list? In-Reply-To: References: <20120914161147.58f58ab8@dino> <20120914205458.09da1b0e@dino> Message-ID: On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote: > Isn't there a case for converting to/from ctypes pointers rather than > capsules? And if capsules, what would the secret word be? Hmm... +1, ctypes would be even better. It's not in the standard library 'till 2.5, but I don't think that's a huge blocker as it can be installed earlier. > Robert Bradshaw wrote: >> >> Given the CObject vs Capsule differences in Py2 vs Py3, and the >> clumsiness in using them (see below), any thoughts on automatically >> converting void* to/from a Python object? There is the sticky issue of >> pointer lifetime management, but we could assume the lifetime is >> managed entirely in C in most cases. >> >> Note that object already has a specific meaning that we can't >> hijack, but perhaps non-void* would be fine to do implicitly? Also, if >> we built this into the language, could we provide type safety via the >> name/description attribute? Something like this could be really handy >> to have and isn't as easily done as a library (especially given the >> 2/3 differences). >> >> - Robert >> >> >> ---------- Forwarded message ---------- >> From: Robert Bradshaw >> Date: Fri, Sep 14, 2012 at 1:11 PM >> Subject: Re: [cython-users] can pointers be stored in Python's dict or >> list? >> To: Mark Summerfield >> Cc: cython-users at googlegroups.com >> >> >> On Fri, Sep 14, 2012 at 12:54 PM, Mark Summerfield >> wrote: >> > Hi Robert, >> > >> > On Fri, 14 Sep 2012 10:45:06 -0700 >> > Robert Bradshaw wrote: >> >> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield >> >> wrote: >> >> > Hi, >> >> > >> >> > (1) >> >> > >> >> > I am using Cython to provide a Python wrapper to a C library. >> >> > The C library has a function which gives a handle (a pointer) which >> >> > is >> >> > then used to call other functions. >> >> > >> >> > I want to be able to store a dictionary of name-pointer pairs so that >> >> > I >> >> > can resuse handles. >> >> >> > >> >> > This doesn't seem to work with dict or list because of the >> >> > Cannot convert 'Thing*' to Python object >> >> > error. >> >> > >> >> > Right now I have a static array and a dictionary: >> >> > >> >> > cdef Thing* pointers[30] >> >> > indexForThingName = {} >> >> > >> >> > but I'd really like to be able to store an arbitrary number of >> >> > pointers, ideally in a dict. >> >> > >> >> > Is this possible? >> >> >> >> Sounds like you want >> >> http://docs.python.org/release/3.1.5/c-api/capsule.html >> > >> > But that means I have to drop down to C. I want to stay in Python and >> > Cython. >> >> You're already "in C" if you're dealing with pointers. >> >> > I was hoping that Cython offered some kind of >> wrapper type, >> > something that at the Python level would look like: >> > >> > class Pointer(object): >> > def __init__(self, void *pointer): >> > cdef void *self.pointer = pointer # Doesn't work >> > >> > that could be put in a dict or list. Then I could do things like this: >> > >> > Pointer p = c_func_that_returns_a_pointer() >> > mylist.append(p) >> > ... >> > Foo *q = mylist.pop() >> >> How about (untested) >> >> from cpython.cobject cimport PyCObject_FromVoidPtr, >> PyCObject_AsVoidPtr >> >> cdef void* p = c_func_that_returns_a_pointer() >> mylist.append(PyCObject_FromVoidPtr(p, NULL) >> ... >> Foo *q = PyCObject_AsVoidPtr(mylist.pop()) >> >> This should work for Python 2, use capsules (at the provided link) for >> Python 3+. For a portable solution, you could cast the void* to a >> intptr_t (or long long, if that's sufficient) which converts back and >> forth from Python just fine (but is more hackish). >> >> > Oh well, I'll see if I can use libcpp's map... >> > >> >> > (2) >> >> > >> >> > Also, I tried doing >> >> > >> >> > from libc.stdio cimport printf >> >> > >> >> > and then in my code: >> >> > >> >> > printf("%p", pointer) >> >> > >> >> > but although it compiled without error, nothing was printed. >> >> >> >> Did it perhaps not flush? You also might want to try adding a newline. >> > >> > I changed it to printf("%p\n", pointer) and now I get >> > Cannot convert 'void *' to Python object >> >> Are you still cimporting printf? >> >> - Robert >> ________________________________ >> >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Sat Sep 15 07:39:38 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 15 Sep 2012 07:39:38 +0200 Subject: [Cython] Fwd: [cython-users] can pointers be stored in Python's dict or list? In-Reply-To: References: <20120914161147.58f58ab8@dino> <20120914205458.09da1b0e@dino> Message-ID: <5054149A.8040009@behnel.de> Robert Bradshaw, 15.09.2012 00:39: > On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote: >> Isn't there a case for converting to/from ctypes pointers rather than >> capsules? And if capsules, what would the secret word be? Hmm... > > +1, ctypes would be even better. It's not in the standard library > 'till 2.5, but I don't think that's a huge blocker as it can be > installed earlier. I'm not entirely sure I see the use case for starting to support the ctypes type system. Is there more to it than just passing pointers through Python code? A capsule (or even a special Cython extension type) sounds like a better option. For example, it's rather unlikely that non-Cython user code will start to support passing in ctypes types, so it would rather be a Cython-to-Cython-only thing anyway. Do you assume that users would want to access C internals through the values that Cython returns to them? That sounds risky. >> Robert Bradshaw wrote: >>> Given the CObject vs Capsule differences in Py2 vs Py3, and the >>> clumsiness in using them (see below), any thoughts on automatically >>> converting void* to/from a Python object? There is the sticky issue of >>> pointer lifetime management, but we could assume the lifetime is >>> managed entirely in C in most cases. >>> >>> Note that object already has a specific meaning that we can't >>> hijack, but perhaps non-void* would be fine to do implicitly? Also, if >>> we built this into the language, could we provide type safety via the >>> name/description attribute? Something like this could be really handy >>> to have and isn't as easily done as a library (especially given the >>> 2/3 differences). We could support both use cases by adding both types to the type system. For example: from cython import capsule cap = some_ptr or, likely nicer: cap = capsule(some_ptr, description) with auto-fallback to CObject on older Py2 versions. I'm not sure about the way back. Maybe a function like "decapsule(cap, description)" would work here? As for ctypes, we might get away with providing a generic cast and do the type matching ourselves for the forward way. For the way back into Cython types, however, we should require an explicitly typed ctypes variable. Just allowing a cast from anything to a Cython C type would let our type conversion code explode. Not sure how much work it would be to implement this, though. The way from Cython to ctypes may just be a simple mapping of type names, whereas the other way might require some utility code overhead. Stefan From markflorisson88 at gmail.com Sat Sep 15 11:42:13 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 15 Sep 2012 10:42:13 +0100 Subject: [Cython] Fwd: [cython-users] can pointers be stored in Python's dict or list? In-Reply-To: <5054149A.8040009@behnel.de> References: <20120914161147.58f58ab8@dino> <20120914205458.09da1b0e@dino> <5054149A.8040009@behnel.de> Message-ID: On 15 September 2012 06:39, Stefan Behnel wrote: > Robert Bradshaw, 15.09.2012 00:39: >> On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote: >>> Isn't there a case for converting to/from ctypes pointers rather than >>> capsules? And if capsules, what would the secret word be? Hmm... >> >> +1, ctypes would be even better. It's not in the standard library >> 'till 2.5, but I don't think that's a huge blocker as it can be >> installed earlier. > > I'm not entirely sure I see the use case for starting to support the ctypes > type system. Is there more to it than just passing pointers through Python > code? A capsule (or even a special Cython extension type) sounds like a > better option. For example, it's rather unlikely that non-Cython user code > will start to support passing in ctypes types, so it would rather be a > Cython-to-Cython-only thing anyway. Actually, I am working on Numba, in which I get ctypes pointers and pointers as integers. It's a usecase, but I'm not sure it should be special-cased, although I do like the automatic type safety it would bring. It's not a big issue for me, I just make Cython accept a Py_uintptr_t and pass in an integer, and I convert that to a (function) pointer using x, which is simple enough (but provides zero type-safety). > Do you assume that users would want to access C internals through the > values that Cython returns to them? That sounds risky. > > >>> Robert Bradshaw wrote: >>>> Given the CObject vs Capsule differences in Py2 vs Py3, and the >>>> clumsiness in using them (see below), any thoughts on automatically >>>> converting void* to/from a Python object? There is the sticky issue of >>>> pointer lifetime management, but we could assume the lifetime is >>>> managed entirely in C in most cases. >>>> >>>> Note that object already has a specific meaning that we can't >>>> hijack, but perhaps non-void* would be fine to do implicitly? Also, if >>>> we built this into the language, could we provide type safety via the >>>> name/description attribute? Something like this could be really handy >>>> to have and isn't as easily done as a library (especially given the >>>> 2/3 differences). > > We could support both use cases by adding both types to the type system. > For example: > > from cython import capsule > > cap = some_ptr > > or, likely nicer: > > cap = capsule(some_ptr, description) > > with auto-fallback to CObject on older Py2 versions. I'm not sure about the > way back. Maybe a function like "decapsule(cap, description)" would work here? > > As for ctypes, we might get away with providing a generic cast and > do the type matching ourselves for the forward way. For the way back into > Cython types, however, we should require an explicitly typed ctypes > variable. Just allowing a cast from anything to a Cython C type would let > our type conversion code explode. > > Not sure how much work it would be to implement this, though. The way from > Cython to ctypes may just be a simple mapping of type names, whereas the > other way might require some utility code overhead. Yeah, it sounds like buffer format string parsing all over again :) Maybe you can build at runtime the Cython type you expect and compare it with the object's _type_? I agree it's probably not trivial, and the use case not common enough. > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From lists at onerussian.com Sun Sep 16 22:48:54 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sun, 16 Sep 2012 16:48:54 -0400 Subject: [Cython] Cython 0.17 final released In-Reply-To: <5041B231.7020905@behnel.de> References: <5041B231.7020905@behnel.de> Message-ID: <20120916204854.GE5955@onerussian.com> On Sat, 01 Sep 2012, Stefan Behnel wrote: > Release notes: http://wiki.cython.org/ReleaseNotes-0.17 http://wiki.cython.org/ReleaseHistory references only 0.16 for me ;-) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From sarvilive at gmail.com Mon Sep 17 09:01:44 2012 From: sarvilive at gmail.com (Saravanan Shanmugham) Date: Mon, 17 Sep 2012 00:01:44 -0700 Subject: [Cython] new FFI library for Python Message-ID: I have been using CFFI for wrapping a whole bunch of C libraries and I can tell you its way easier than Cython. And a couple of clarifications of misconceptions mentioned on this list. 1. It does not "require" GCC at run time. The extension can be compiled at package/install time using the GCC and can run without need GCC at run time. 2. Its way easier to wrap C libraries than Cython. I would know, since I started with Cython. And after many email back and forth with the cython-users mailing list, I was referred to CFFI which turned out great. The trouble was wrapping C structures and datastructures with Python wrappers. 3. I had to just copy over the C header files as is and had a replace a few #define values with "..." to ask it to automatically get these values during compile time from the compiler. This makes it even more simpler. Now, I am still a Cython lover when it comes optimizing portions of my Python code and having it compiled to C. The question I had in mind is, If I had say a shared math library libmath.so and wrapped it with a CFFI based python wrapper, which includes a portion of python wrapper code that calls into the generated Python/C wrapper shared library. And then wrote a piece of cython code that invoked that CFFI based python wrapper, how will my performance be compared to writing the python wrapper for libmath.so with cython cdef or cpdef wrapper function definitions. Thanks, Sarvi -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Mon Sep 17 09:16:07 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 17 Sep 2012 09:16:07 +0200 Subject: [Cython] new FFI library for Python In-Reply-To: References: Message-ID: <940b1366-46d8-49c0-9322-f7a6015fe1d3@email.android.com> Saravanan Shanmugham wrote: >I have been using CFFI for wrapping a whole bunch of C libraries and I >can >tell you its way easier than Cython. > >And a couple of clarifications of misconceptions mentioned on this >list. >1. It does not "require" GCC at run time. The extension can be compiled >at package/install time using the GCC and can run without need GCC at >run >time. > 2. Its way easier to wrap C libraries than Cython. I would know, since >I started with Cython. And after many email back and forth with the >cython-users mailing list, I was referred to CFFI which turned out >great. >The trouble was wrapping C structures and datastructures with Python >wrappers. >3. I had to just copy over the C header files as is and had a replace a >few #define values with "..." to ask it to automatically get these >values >during compile time from the compiler. This makes it even more >simpler. > > >Now, I am still a Cython lover when it comes optimizing portions of my >Python code and having it compiled to C. > >The question I had in mind is, If I had say a shared math library >libmath.so and wrapped it with a CFFI based python wrapper, which >includes >a portion of python wrapper code that calls into the generated Python/C >wrapper shared library. >And then wrote a piece of cython code that invoked that CFFI based >python >wrapper, how will my performance be compared to >writing the python wrapper for libmath.so with cython cdef or cpdef >wrapper function definitions. It will be worse, it will be like calling Python functions rather than C. Fixing this can be done but would require support in both cffi and Cython. We were discussing a wrapper-language-neutral protocol for fixing this problem (also so that Cython could call numpy.sin and friends without any overhead). CEP1000 is an early draft of that idea; it has developed a lot since then but the details are in mailing list threads and my and Robert's heads. If anybody would like to work on that then I'll find the time to write down all the ideas me and Robert worked out. (but unless anybody stands ready I'd rather wait another month) Dag Sverre > >Thanks, >Sarvi >_______________________________________________ >cython-devel mailing list >cython-devel at python.org >http://mail.python.org/mailman/listinfo/cython-devel -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. From robertwb at gmail.com Mon Sep 17 23:11:31 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 17 Sep 2012 14:11:31 -0700 Subject: [Cython] Cython 0.17 final released In-Reply-To: <20120916204854.GE5955@onerussian.com> References: <5041B231.7020905@behnel.de> <20120916204854.GE5955@onerussian.com> Message-ID: On Sun, Sep 16, 2012 at 1:48 PM, Yaroslav Halchenko wrote: > > On Sat, 01 Sep 2012, Stefan Behnel wrote: >> Release notes: http://wiki.cython.org/ReleaseNotes-0.17 > > http://wiki.cython.org/ReleaseHistory > references only 0.16 for me > ;-) Fixed. From cwg at falma.de Tue Sep 18 16:19:05 2012 From: cwg at falma.de (Christoph Groth) Date: Tue, 18 Sep 2012 16:19:05 +0200 Subject: [Cython] [PATCH] make memoryviews work when strides is NULL (unfinished) Message-ID: <87zk4n4ebp.fsf@falma.de> Hi, Thanks a lot for the recently added generic memoryview support, it's just what we needed to optimize an important part of our program. However, I believe that there's a problem with Cython's support for objects which do not provide "strides" information (because they are C-contiguous). The buffer API documentation [1] is a bit ambiguous on this, but it says "If strides is NULL, the array is interpreted as a standard n-dimensional C-array." In any case, numpy happily accepts such buffers and computes the "strides" itself. A thread [2] on Python's devel mailing list also supports the view that setting strides to NULL is OK even if strides were requested. You might ask: why not simply provide strides when they have been requested? The problem is to find space for them. malloc'ing or PyMem_Alloc'ing memory for strides each time when a buffer is requested is slow, and reserving space for strides within the object is wasteful when "shape" is all that's needed and you have _many_ small objects which expose the buffer interface. The provided patch works for our use case, but someone with a good understanding of Cython's memoryview support should finish it. I believe the patch will break more complicated buffer use cases. [1] http://docs.python.org/dev/c-api/buffer.html [2] http://thread.gmane.org/gmane.comp.python.devel/134387 Thank you for considering fixing this issue in Cython, Christoph --- Cython/Utility/MemoryView_C.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/Cython/Utility/MemoryView_C.c b/Cython/Utility/MemoryView_C.c index 84e55da..c2524b0 100644 --- a/Cython/Utility/MemoryView_C.c +++ b/Cython/Utility/MemoryView_C.c @@ -207,12 +207,6 @@ static int __Pyx_ValidateAndInit_memviewslice( goto fail; } - if (!buf->strides) { - PyErr_SetString(PyExc_ValueError, - "buffer does not supply strides necessary for memoryview."); - goto fail; - } - for(i=0; istrides) { + for (i = 0; i < ndim; i++) { + memviewslice->strides[i] = buf->strides[i]; + } + } else { + Py_ssize_t stride = buf->itemsize; + for (i = ndim - 1; i >= 0; i--) { + memviewslice->strides[i] = stride; + stride *= buf->shape[i]; + } + } + for (i = 0; i < ndim; i++) { - memviewslice->strides[i] = buf->strides[i]; memviewslice->shape[i] = buf->shape[i]; if (buf->suboffsets) { memviewslice->suboffsets[i] = buf->suboffsets[i]; -- 1.7.10.4 From cwg at falma.de Tue Sep 18 21:55:16 2012 From: cwg at falma.de (Christoph Groth) Date: Tue, 18 Sep 2012 21:55:16 +0200 Subject: [Cython] read-only memory views Message-ID: <87txuvgl7f.fsf@falma.de> Hello, I have written a python extension module, tinyarray, (to be made public soon) which implements an important subset of numpy optimized for _small_ arrays. Tinyarrays are immutable, which makes them usable as dictionary keys. Some important operations, i.e. creation of a small array from a tuple or the scalar product of two small arrays are 10 to 45 times faster than with numpy. This can really boost the performance of programs which use small numpy arrays. (I know that numpy is meant to be used for large arrays, but there are legitimate uses of small arrays as well.) It would be great if Cython could support tinyarrays. Right now, it doesn't. See http://article.gmane.org/gmane.comp.python.cython.user/7329 (As a temporary workaround, I have to made the buffers writable. But this breaks strict immutability.) Would fixing this be a lot of work? Cheers, Christoph From markflorisson88 at gmail.com Wed Sep 19 00:42:39 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 18 Sep 2012 23:42:39 +0100 Subject: [Cython] read-only memory views In-Reply-To: <87txuvgl7f.fsf@falma.de> References: <87txuvgl7f.fsf@falma.de> Message-ID: On 18 September 2012 20:55, Christoph Groth wrote: > Hello, > > I have written a python extension module, tinyarray, (to be made public > soon) which implements an important subset of numpy optimized for > _small_ arrays. Tinyarrays are immutable, which makes them usable as > dictionary keys. > > Some important operations, i.e. creation of a small array from a tuple > or the scalar product of two small arrays are 10 to 45 times faster than > with numpy. This can really boost the performance of programs which use > small numpy arrays. (I know that numpy is meant to be used for large > arrays, but there are legitimate uses of small arrays as well.) > > It would be great if Cython could support tinyarrays. Right now, it > doesn't. See > http://article.gmane.org/gmane.comp.python.cython.user/7329 > > (As a temporary workaround, I have to made the buffers writable. But > this breaks strict immutability.) > > Would fixing this be a lot of work? That depends on how we implement it. A 'const' keyword has been discussed in general. This would be a little more work, since it needs grammar modifications. Also, there would be a difference between declaring the data and the variable const, i.e. 'const double[:]' vs 'double[:] const', which I personally find bothersome. Using flags are obvious, like the ones in cython.view, i.e. double[:view.readonly], but the problem with that is which axis to choose (or any axis?), and we don't have a way of combining flags. So we would not only need view.contiguous, but also view.contiguous_readonly etc. Without a greater plan for 'const' I think it might be awkward to use for this purpose. I think the flags are apt, and should be used in the first dimension. If we wait for other people to comment we could give you pointers if you want to give implementing it a swing, it wouldn't be too hard. > Cheers, > Christoph > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Wed Sep 19 00:44:51 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 18 Sep 2012 23:44:51 +0100 Subject: [Cython] [PATCH] make memoryviews work when strides is NULL (unfinished) In-Reply-To: <87zk4n4ebp.fsf@falma.de> References: <87zk4n4ebp.fsf@falma.de> Message-ID: On 18 September 2012 15:19, Christoph Groth wrote: > Hi, > > Thanks a lot for the recently added generic memoryview support, it's > just what we needed to optimize an important part of our program. > > However, I believe that there's a problem with Cython's support for > objects which do not provide "strides" information (because they are > C-contiguous). > > The buffer API documentation [1] is a bit ambiguous on this, but it says > > "If strides is NULL, the array is interpreted as a standard > n-dimensional C-array." > > In any case, numpy happily accepts such buffers and computes the > "strides" itself. > > A thread [2] on Python's devel mailing list also supports the view that > setting strides to NULL is OK even if strides were requested. > > You might ask: why not simply provide strides when they have been > requested? The problem is to find space for them. malloc'ing or > PyMem_Alloc'ing memory for strides each time when a buffer is requested > is slow, and reserving space for strides within the object is wasteful > when "shape" is all that's needed and you have _many_ small objects > which expose the buffer interface. > > The provided patch works for our use case, but someone with a good > understanding of Cython's memoryview support should finish it. I > believe the patch will break more complicated buffer use cases. > > [1] http://docs.python.org/dev/c-api/buffer.html > [2] http://thread.gmane.org/gmane.comp.python.devel/134387 > > Thank you for considering fixing this issue in Cython, > > Christoph > > > --- > Cython/Utility/MemoryView_C.c | 19 ++++++++++++------- > 1 file changed, 12 insertions(+), 7 deletions(-) > > diff --git a/Cython/Utility/MemoryView_C.c b/Cython/Utility/MemoryView_C.c > index 84e55da..c2524b0 100644 > --- a/Cython/Utility/MemoryView_C.c > +++ b/Cython/Utility/MemoryView_C.c > @@ -207,12 +207,6 @@ static int __Pyx_ValidateAndInit_memviewslice( > goto fail; > } > > - if (!buf->strides) { > - PyErr_SetString(PyExc_ValueError, > - "buffer does not supply strides necessary for memoryview."); > - goto fail; > - } > - > for(i=0; i spec = axes_specs[i]; > > @@ -320,8 +314,19 @@ __Pyx_init_memviewslice(struct __pyx_memoryview_obj *memview, > goto fail; > } > > + if (buf->strides) { > + for (i = 0; i < ndim; i++) { > + memviewslice->strides[i] = buf->strides[i]; > + } > + } else { > + Py_ssize_t stride = buf->itemsize; > + for (i = ndim - 1; i >= 0; i--) { > + memviewslice->strides[i] = stride; > + stride *= buf->shape[i]; > + } > + } > + > for (i = 0; i < ndim; i++) { > - memviewslice->strides[i] = buf->strides[i]; > memviewslice->shape[i] = buf->shape[i]; > if (buf->suboffsets) { > memviewslice->suboffsets[i] = buf->suboffsets[i]; > -- > 1.7.10.4 > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Hey Christoph, Thanks for the patch, it looks fine to me. If you want, you could make a pull request and I'll merge it. Mark From robertwb at gmail.com Wed Sep 19 01:15:43 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 18 Sep 2012 16:15:43 -0700 Subject: [Cython] read-only memory views In-Reply-To: References: <87txuvgl7f.fsf@falma.de> Message-ID: On Tue, Sep 18, 2012 at 3:42 PM, mark florisson wrote: > On 18 September 2012 20:55, Christoph Groth wrote: >> Hello, >> >> I have written a python extension module, tinyarray, (to be made public >> soon) which implements an important subset of numpy optimized for >> _small_ arrays. Tinyarrays are immutable, which makes them usable as >> dictionary keys. >> >> Some important operations, i.e. creation of a small array from a tuple >> or the scalar product of two small arrays are 10 to 45 times faster than >> with numpy. This can really boost the performance of programs which use >> small numpy arrays. (I know that numpy is meant to be used for large >> arrays, but there are legitimate uses of small arrays as well.) >> >> It would be great if Cython could support tinyarrays. Right now, it >> doesn't. See >> http://article.gmane.org/gmane.comp.python.cython.user/7329 >> >> (As a temporary workaround, I have to made the buffers writable. But >> this breaks strict immutability.) >> >> Would fixing this be a lot of work? > > That depends on how we implement it. A 'const' keyword has been > discussed in general. This would be a little more work, since it needs > grammar modifications. Also, there would be a difference between > declaring the data and the variable const, i.e. 'const double[:]' vs > 'double[:] const', which I personally find bothersome. > > Using flags are obvious, like the ones in cython.view, i.e. > double[:view.readonly], but the problem with that is which axis to > choose (or any axis?), and we don't have a way of combining flags. So > we would not only need view.contiguous, but also > view.contiguous_readonly etc. I'd much rather be able to write view.readonly & view.contiguous than come up with flags specifying the various permutations. > Without a greater plan for 'const' I think it might be awkward to use > for this purpose. I think the flags are apt, and should be used in the > first dimension. If we wait for other people to comment we could give > you pointers if you want to give implementing it a swing, it wouldn't > be too hard. Coincidentally, I actually implemented rudimentary const support a couple of days ago. It's not perfect, but at least it allows you to provide the correct declarations to deal with const polluted^H^H^H^H correct code (including const c++ class methods, for which there was simply no workaround). I just need to write some tests before I feel comfortable pushing it. That said, I think a flag is a good fit here, though const could work as well. - Robert From cwg at falma.de Wed Sep 19 09:03:41 2012 From: cwg at falma.de (Christoph Groth) Date: Wed, 19 Sep 2012 09:03:41 +0200 Subject: [Cython] [PATCH] make memoryviews work when strides is NULL (unfinished) References: <87zk4n4ebp.fsf@falma.de> Message-ID: <87pq5ih4tu.fsf_-_@falma.de> Hi Mark, > Thanks for the patch, it looks fine to me. If you want, you could make > a pull request and I'll merge it. Ok, the pull request is here: https://github.com/cython/cython/pull/150 Please double-check that the dereferencing of buf->strides in PyErr_SetString(PyExc_ValueError is still safe. I lack the understanding of Cython's memoryview support to do this myself. Christoph From cwg at falma.de Wed Sep 19 09:06:55 2012 From: cwg at falma.de (Christoph Groth) Date: Wed, 19 Sep 2012 09:06:55 +0200 Subject: [Cython] read-only memory views References: <87txuvgl7f.fsf@falma.de> Message-ID: <87lig6h4og.fsf@falma.de> Robert Bradshaw writes: > Coincidentally, I actually implemented rudimentary const support a > couple of days ago. It's not perfect, but at least it allows you to > provide the correct declarations to deal with const polluted^H^H^H^H > correct code (including const c++ class methods, for which there was > simply no workaround). I just need to write some tests before I feel > comfortable pushing it. > > That said, I think a flag is a good fit here, though const could work > as well. So let's wait for Robert's const support and re-examine the situation when it's there. Christoph From markflorisson88 at gmail.com Wed Sep 19 11:48:08 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 19 Sep 2012 10:48:08 +0100 Subject: [Cython] read-only memory views In-Reply-To: References: <87txuvgl7f.fsf@falma.de> Message-ID: On 19 September 2012 00:15, Robert Bradshaw wrote: > On Tue, Sep 18, 2012 at 3:42 PM, mark florisson > wrote: >> On 18 September 2012 20:55, Christoph Groth wrote: >>> Hello, >>> >>> I have written a python extension module, tinyarray, (to be made public >>> soon) which implements an important subset of numpy optimized for >>> _small_ arrays. Tinyarrays are immutable, which makes them usable as >>> dictionary keys. >>> >>> Some important operations, i.e. creation of a small array from a tuple >>> or the scalar product of two small arrays are 10 to 45 times faster than >>> with numpy. This can really boost the performance of programs which use >>> small numpy arrays. (I know that numpy is meant to be used for large >>> arrays, but there are legitimate uses of small arrays as well.) >>> >>> It would be great if Cython could support tinyarrays. Right now, it >>> doesn't. See >>> http://article.gmane.org/gmane.comp.python.cython.user/7329 >>> >>> (As a temporary workaround, I have to made the buffers writable. But >>> this breaks strict immutability.) >>> >>> Would fixing this be a lot of work? >> >> That depends on how we implement it. A 'const' keyword has been >> discussed in general. This would be a little more work, since it needs >> grammar modifications. Also, there would be a difference between >> declaring the data and the variable const, i.e. 'const double[:]' vs >> 'double[:] const', which I personally find bothersome. >> >> Using flags are obvious, like the ones in cython.view, i.e. >> double[:view.readonly], but the problem with that is which axis to >> choose (or any axis?), and we don't have a way of combining flags. So >> we would not only need view.contiguous, but also >> view.contiguous_readonly etc. > > I'd much rather be able to write view.readonly & view.contiguous than > come up with flags specifying the various permutations. Dag and I discussed it last year and decided to go for the combinations, e.g. indirect_contiguous instead of indirect & contiguous. It looks a lot like flags, so perhaps it should use |. Either way I'm not too fond of the flags. >> Without a greater plan for 'const' I think it might be awkward to use >> for this purpose. I think the flags are apt, and should be used in the >> first dimension. If we wait for other people to comment we could give >> you pointers if you want to give implementing it a swing, it wouldn't >> be too hard. > > Coincidentally, I actually implemented rudimentary const support a > couple of days ago. It's not perfect, but at least it allows you to > provide the correct declarations to deal with const polluted^H^H^H^H > correct code (including const c++ class methods, for which there was > simply no workaround). I just need to write some tests before I feel > comfortable pushing it. > > That said, I think a flag is a good fit here, though const could work as well. Oh that's great, either would be good, but if we have const then using that for memoryviews would be more consistent. Does this distinguish between const * and * const? > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Wed Sep 19 11:52:43 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 19 Sep 2012 10:52:43 +0100 Subject: [Cython] [PATCH] make memoryviews work when strides is NULL (unfinished) In-Reply-To: <87pq5ih4tu.fsf_-_@falma.de> References: <87zk4n4ebp.fsf@falma.de> <87pq5ih4tu.fsf_-_@falma.de> Message-ID: On 19 September 2012 08:03, Christoph Groth wrote: > Hi Mark, > >> Thanks for the patch, it looks fine to me. If you want, you could make >> a pull request and I'll merge it. > > Ok, the pull request is here: > https://github.com/cython/cython/pull/150 > > Please double-check that the dereferencing of buf->strides in > PyErr_SetString(PyExc_ValueError is still safe. I lack the > understanding of Cython's memoryview support to do this myself. > > Christoph Yeah, I think it needs a little more work. I'll see if I have some time tonight, thanks! > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From robertwb at gmail.com Wed Sep 19 16:49:28 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 19 Sep 2012 07:49:28 -0700 Subject: [Cython] read-only memory views In-Reply-To: <87lig6h4og.fsf@falma.de> References: <87txuvgl7f.fsf@falma.de> <87lig6h4og.fsf@falma.de> Message-ID: On Wed, Sep 19, 2012 at 12:06 AM, Christoph Groth wrote: > Robert Bradshaw > writes: > >> Coincidentally, I actually implemented rudimentary const support a >> couple of days ago. It's not perfect, but at least it allows you to >> provide the correct declarations to deal with const polluted^H^H^H^H >> correct code (including const c++ class methods, for which there was >> simply no workaround). I just need to write some tests before I feel >> comfortable pushing it. >> >> That said, I think a flag is a good fit here, though const could work >> as well. > > So let's wait for Robert's const support and re-examine the situation > when it's there. https://github.com/cython/cython/commit/8c4a629904de8c21cd655bb63d8e1ea1ab89142f From ian.h.bell at gmail.com Thu Sep 20 21:11:47 2012 From: ian.h.bell at gmail.com (Ian Bell) Date: Thu, 20 Sep 2012 15:11:47 -0400 Subject: [Cython] Auto-pickle progress? Message-ID: There is a proposal for auto-pickling of Cython extension types: http://wiki.cython.org/enhancements/pickling I was wondering whether there has been any progress on this front? Auto-pickling would be tremendously helpful as pickling and unpickling is one of the most annoying features of working with threads and processes in python. Regards, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Thu Sep 20 22:34:17 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 20 Sep 2012 13:34:17 -0700 Subject: [Cython] Auto-pickle progress? In-Reply-To: References: Message-ID: On Thu, Sep 20, 2012 at 12:11 PM, Ian Bell wrote: > There is a proposal for auto-pickling of Cython extension types: > http://wiki.cython.org/enhancements/pickling > > I was wondering whether there has been any progress on this front? > > Auto-pickling would be tremendously helpful as pickling and unpickling is > one of the most annoying features of working with threads and processes in > python. Though I agree this would be really useful, some people are less enthusiastic. In any case, no work has been done on this front, but we accept contributions. - Robert From stefan_ml at behnel.de Sat Sep 22 15:03:11 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 22 Sep 2012 15:03:11 +0200 Subject: [Cython] cpdef, nogil and void return (was: [cython-users] Use the function abs in a cpdef function with nogil) In-Reply-To: References: <397b9745-6923-4bf9-851f-58985978341a@googlegroups.com> <1155087224578582887@unknownmsgid> Message-ID: <505DB70F.2060402@behnel.de> [CC-ing cython-devel here as this is a design issue] Joaquin Cuenca Abela, 21.09.2012 19:32: > On Fri, Sep 21, 2012 at 6:12 PM, Chris Barker wrote: >> On Fri, Sep 21, 2012 at 2:48 AM, Joaquin Cuenca Abela wrote: >>> I don't set a return type, it gives the error: >>> >>> cpdef undo_filter_sub(int filter_unit, unsigned char[:] scanline, >>> ^ >>> ------------------------------------------------------------ >>> cpngfilters.pyx:10:6: Function with Python return type cannot be declared nogil >> >> cpdef means build both a pyton function and a C function -- C >> functions need a return type. Cython defaulats to a generic Python >> object as a return type, which could cause issue with nogil, so it >> won't let you do that -- so try declareing a void return type: >> >> cpdef void undo_filter_sub(int filter_unit, unsigned char[:] scanline, >> >> I suspect that will give you None for the Python version, but I'm just >> guessing here. >> >> If that doesn't work, use int or something, and ignore the return value. > > nope, with void I was getting "Cannot convert 'void' to Python > object". What I did is put int and return 0, maybe this should be > added to the docs. I think we should support "cpdef void func()" and just let the Python wrapper return None automatically. That's the expected Python equivalent of returning nothing. Shouldn't be hard to implement. cpdef functions that are also declared nogil are somewhat unusual, but I don't see them being outright wrong. Basically, it would allow you to call them from Cython code with or without holding the GIL as well as directly from Python code (that always holds the GIL). Might be handy for avoid code redundancy in some cases. Patches welcome, as always. Note, however, that declaring the return type as void will make exception handling more difficult. That's ok for a nogil function, which cannot raise exceptions, but it's worth remembering for normal functions. Stefan From markflorisson88 at gmail.com Sat Sep 22 23:43:02 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 22 Sep 2012 22:43:02 +0100 Subject: [Cython] cpdef, nogil and void return (was: [cython-users] Use the function abs in a cpdef function with nogil) In-Reply-To: <505DB70F.2060402@behnel.de> References: <397b9745-6923-4bf9-851f-58985978341a@googlegroups.com> <1155087224578582887@unknownmsgid> <505DB70F.2060402@behnel.de> Message-ID: On 22 September 2012 14:03, Stefan Behnel wrote: > [CC-ing cython-devel here as this is a design issue] > > Joaquin Cuenca Abela, 21.09.2012 19:32: >> On Fri, Sep 21, 2012 at 6:12 PM, Chris Barker wrote: >>> On Fri, Sep 21, 2012 at 2:48 AM, Joaquin Cuenca Abela wrote: >>>> I don't set a return type, it gives the error: >>>> >>>> cpdef undo_filter_sub(int filter_unit, unsigned char[:] scanline, >>>> ^ >>>> ------------------------------------------------------------ >>>> cpngfilters.pyx:10:6: Function with Python return type cannot be declared nogil >>> >>> cpdef means build both a pyton function and a C function -- C >>> functions need a return type. Cython defaulats to a generic Python >>> object as a return type, which could cause issue with nogil, so it >>> won't let you do that -- so try declareing a void return type: >>> >>> cpdef void undo_filter_sub(int filter_unit, unsigned char[:] scanline, >>> >>> I suspect that will give you None for the Python version, but I'm just >>> guessing here. >>> >>> If that doesn't work, use int or something, and ignore the return value. >> >> nope, with void I was getting "Cannot convert 'void' to Python >> object". What I did is put int and return 0, maybe this should be >> added to the docs. > > I think we should support "cpdef void func()" and just let the Python > wrapper return None automatically. That's the expected Python equivalent of > returning nothing. Shouldn't be hard to implement. Sounds like a good idea, that will make it easier for people. > cpdef functions that are also declared nogil are somewhat unusual, but I > don't see them being outright wrong. Basically, it would allow you to call > them from Cython code with or without holding the GIL as well as directly > from Python code (that always holds the GIL). Might be handy for avoid code > redundancy in some cases. > > Patches welcome, as always. > > Note, however, that declaring the return type as void will make exception > handling more difficult. That's ok for a nogil function, which cannot raise > exceptions, but it's worth remembering for normal functions. Nogil functions can raise exceptions without problem. It would be nice to implement the Cython abi at some point where we get rid of unraisable exceptions at the Cython and Python level. > Stefan > From sturla at molden.no Tue Sep 25 12:44:04 2012 From: sturla at molden.no (Sturla Molden) Date: Tue, 25 Sep 2012 12:44:04 +0200 Subject: [Cython] Auto-pickle progress? In-Reply-To: References: Message-ID: <50618AF4.6090104@molden.no> On 20.09.2012 21:11, Ian Bell wrote: > Auto-pickling would be tremendously helpful as pickling and unpickling > is one of the most annoying features of working with threads and > processes in python. How should Cython interfere how to pickle a C pointer? cdef class foobar: cdef double *data A C object can be anything. Cython does not know anything about size, offset or strides, or even if it's safe to take a copy. Example: How to pickle a shared memory buffer? Surely we cannot take a copy, because that would defeat the purpose of "shared" memory. And even if could take a copy, how many bytes should be copied? Do you think an autopickler could have figured this out? https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_sysv.pyx https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_win.pyx On yes, the code is different on Unix and Windows, something the auto-pickler could not possibly know either. Auto-pickling cdef classes is not doable, IMHO. And by the way, implementing a __reduce__ method manually is not very difficult either. Sturla Molden From ian.h.bell at gmail.com Tue Sep 25 14:41:27 2012 From: ian.h.bell at gmail.com (Ian Bell) Date: Tue, 25 Sep 2012 08:41:27 -0400 Subject: [Cython] Auto-pickle progress? In-Reply-To: <50618AF4.6090104@molden.no> References: <50618AF4.6090104@molden.no> Message-ID: I agree that some of the more powerful elements of Cython would be/are difficult or impossible to pickle in a general way. I have a counter use-case where auto-pickling would be very useful. I have cdef classes that have many (think 50-100) members of double and long types, as well as other cdef classes that are also only using Cython-friendly data types. I have implemented pickling for these classes and it sucks. Trying to maintain the list of parameters that need to be pickled is a major hassle. The biggest problem is that if you add add a parameter to the class but forget to add it to the pickling function, you end up with the parameter not getting reloaded on unpickling. This is not good! Yes it is up the programmer to do this right, but I could use some help. What I have ended up doing for my other classes is define a __cdict__ function that returns a dictionary with the values and parameters as defined in the cdef class. In this way, pickling and unpickling is a little bit more pleasant. But it would be fantastic if the __cdict__ could be baked into the class automatically. It would be fine for me if only the things that could be easily handled would be returned, and a second list of attributes that require more work to be serialized so that the user/coder could be sure that all the attributes are being handled and none get forgotten. Also, the docs on pickling of cdef classes is very sparse, a simple example would be very beneficial. The is one in the examples in the source, but it is not easy to find Ian On Tue, Sep 25, 2012 at 6:44 AM, Sturla Molden wrote: > On 20.09.2012 21:11, Ian Bell wrote: > > Auto-pickling would be tremendously helpful as pickling and unpickling >> is one of the most annoying features of working with threads and >> processes in python. >> > > How should Cython interfere how to pickle a C pointer? > > cdef class foobar: > cdef double *data > > A C object can be anything. Cython does not know anything about size, > offset or strides, or even if it's safe to take a copy. > > Example: How to pickle a shared memory buffer? Surely we cannot take a > copy, because that would defeat the purpose of "shared" memory. And even if > could take a copy, how many bytes should be copied? Do you think an > autopickler could have figured this out? > > https://github.com/**sturlamolden/sharedmem-numpy/**blob/master/sharedmem/ > **sharedmemory_sysv.pyx > > https://github.com/**sturlamolden/sharedmem-numpy/**blob/master/sharedmem/ > **sharedmemory_win.pyx > > On yes, the code is different on Unix and Windows, something the > auto-pickler could not possibly know either. > > Auto-pickling cdef classes is not doable, IMHO. > > > And by the way, implementing a __reduce__ method manually is not very > difficult either. > > > Sturla Molden > > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Sep 25 14:55:41 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 25 Sep 2012 14:55:41 +0200 Subject: [Cython] ref-leak in 0.17 Message-ID: <5061A9CD.4040508@behnel.de> Hi, there is a reference leak in the new dict iteration code in 0.17: http://trac.cython.org/cython_trac/ticket/790 I'll see if I can fix it and then release a pure bugfix 0.17.1 right afterwards. Please take a look at your own patch queues to see if there are any other important bugs that you would like to see fixed in that release. Stefan From stefan_ml at behnel.de Tue Sep 25 16:20:35 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 25 Sep 2012 16:20:35 +0200 Subject: [Cython] Cython 0.17 final released In-Reply-To: References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> <50430D67.5050608@behnel.de> Message-ID: <5061BDB3.2020505@behnel.de> Bradley M. Froehle, 11.09.2012 00:34: > Any chance we can tag the official 0.17 release in Github as well? BTW, do we use lightweight tags or annotated tags in git? And why? Stefan From stefan_ml at behnel.de Tue Sep 25 17:12:49 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 25 Sep 2012 17:12:49 +0200 Subject: [Cython] ref-leak in 0.17 In-Reply-To: <5061A9CD.4040508@behnel.de> References: <5061A9CD.4040508@behnel.de> Message-ID: <5061C9F1.5030106@behnel.de> Stefan Behnel, 25.09.2012 14:55: > there is a reference leak in the new dict iteration code in 0.17: > > http://trac.cython.org/cython_trac/ticket/790 > > I'll see if I can fix it and then release a pure bugfix 0.17.1 right > afterwards. > > Please take a look at your own patch queues to see if there are any other > important bugs that you would like to see fixed in that release. Update: I fixed this, so 0.17.1 will be released tomorrow (CEST) unless you've thrown in your veto by then. @Mark: what about the NULL strides changes? Would you consider them safe enough to go in or can they wait a bit? If unsure, please choose the latter. ;) Stefan From markflorisson88 at gmail.com Tue Sep 25 17:27:23 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 25 Sep 2012 16:27:23 +0100 Subject: [Cython] ref-leak in 0.17 In-Reply-To: <5061C9F1.5030106@behnel.de> References: <5061A9CD.4040508@behnel.de> <5061C9F1.5030106@behnel.de> Message-ID: On 25 September 2012 16:12, Stefan Behnel wrote: > Stefan Behnel, 25.09.2012 14:55: >> there is a reference leak in the new dict iteration code in 0.17: >> >> http://trac.cython.org/cython_trac/ticket/790 >> >> I'll see if I can fix it and then release a pure bugfix 0.17.1 right >> afterwards. >> >> Please take a look at your own patch queues to see if there are any other >> important bugs that you would like to see fixed in that release. > > Update: I fixed this, so 0.17.1 will be released tomorrow (CEST) unless > you've thrown in your veto by then. > > @Mark: what about the NULL strides changes? Would you consider them safe > enough to go in or can they wait a bit? If unsure, please choose the latter. ;) > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Hey Stefan, Thanks for asking. Yeah I merged it last weekend with tests and error checking code, it should be fine. Mark From robertwb at gmail.com Tue Sep 25 17:47:02 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 25 Sep 2012 08:47:02 -0700 Subject: [Cython] Auto-pickle progress? In-Reply-To: References: <50618AF4.6090104@molden.no> Message-ID: On Tue, Sep 25, 2012 at 5:41 AM, Ian Bell wrote: > I agree that some of the more powerful elements of Cython would be/are > difficult or impossible to pickle in a general way. I have a counter > use-case where auto-pickling would be very useful. > > I have cdef classes that have many (think 50-100) members of double and long > types, as well as other cdef classes that are also only using > Cython-friendly data types. I have implemented pickling for these classes > and it sucks. Trying to maintain the list of parameters that need to be > pickled is a major hassle. The biggest problem is that if you add add a > parameter to the class but forget to add it to the pickling function, you > end up with the parameter not getting reloaded on unpickling. This is not > good! Yes it is up the programmer to do this right, but I could use some > help. > > What I have ended up doing for my other classes is define a __cdict__ > function that returns a dictionary with the values and parameters as defined > in the cdef class. In this way, pickling and unpickling is a little bit > more pleasant. But it would be fantastic if the __cdict__ could be baked > into the class automatically. It would be fine for me if only the things > that could be easily handled would be returned, and a second list of > attributes that require more work to be serialized so that the user/coder > could be sure that all the attributes are being handled and none get > forgotten. Exactly. Even if we handled only cdef classes with only (transitively) picklable fields, this would go a long way. The only question to have is how to handle changes in the class struct between pickling and unpickling. Eventually we could come up with syntax/mechanisms for manually specifying serialization of more complicated members like double*, but that's the kind of class you might have to write a __reduce__ method on anyways. > Also, the docs on pickling of cdef classes is very sparse, a simple example > would be very beneficial. The is one in the examples in the source, but it > is not easy to find Agreed. > On Tue, Sep 25, 2012 at 6:44 AM, Sturla Molden wrote: >> >> On 20.09.2012 21:11, Ian Bell wrote: >> >>> Auto-pickling would be tremendously helpful as pickling and unpickling >>> is one of the most annoying features of working with threads and >>> processes in python. >> >> >> How should Cython interfere how to pickle a C pointer? >> >> cdef class foobar: >> cdef double *data >> >> A C object can be anything. Cython does not know anything about size, >> offset or strides, or even if it's safe to take a copy. >> >> Example: How to pickle a shared memory buffer? Surely we cannot take a >> copy, because that would defeat the purpose of "shared" memory. And even if >> could take a copy, how many bytes should be copied? Do you think an >> autopickler could have figured this out? >> >> >> https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_sysv.pyx >> >> >> https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_win.pyx >> >> On yes, the code is different on Unix and Windows, something the >> auto-pickler could not possibly know either. >> >> Auto-pickling cdef classes is not doable, IMHO. >> >> >> And by the way, implementing a __reduce__ method manually is not very >> difficult either. >> >> >> Sturla Molden >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Tue Sep 25 17:48:27 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 25 Sep 2012 17:48:27 +0200 Subject: [Cython] ref-leak in 0.17 In-Reply-To: References: <5061A9CD.4040508@behnel.de> <5061C9F1.5030106@behnel.de> Message-ID: <5061D24B.2010500@behnel.de> mark florisson, 25.09.2012 17:27: > On 25 September 2012 16:12, Stefan Behnel wrote: >> Stefan Behnel, 25.09.2012 14:55: >>> there is a reference leak in the new dict iteration code in 0.17: >>> >>> http://trac.cython.org/cython_trac/ticket/790 >>> >>> I'll see if I can fix it and then release a pure bugfix 0.17.1 right >>> afterwards. >>> >>> Please take a look at your own patch queues to see if there are any other >>> important bugs that you would like to see fixed in that release. >> >> Update: I fixed this, so 0.17.1 will be released tomorrow (CEST) unless >> you've thrown in your veto by then. >> >> @Mark: what about the NULL strides changes? Would you consider them safe >> enough to go in or can they wait a bit? If unsure, please choose the latter. ;) > > Thanks for asking. Yeah I merged it last weekend with tests and error > checking code, it should be fine. Ok - it won't impact existing code but fixes the implementation, so I think it can go in. Could you merge it over? Stefan From markflorisson88 at gmail.com Tue Sep 25 22:37:19 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 25 Sep 2012 21:37:19 +0100 Subject: [Cython] ref-leak in 0.17 In-Reply-To: <5061D24B.2010500@behnel.de> References: <5061A9CD.4040508@behnel.de> <5061C9F1.5030106@behnel.de> <5061D24B.2010500@behnel.de> Message-ID: On 25 September 2012 16:48, Stefan Behnel wrote: > mark florisson, 25.09.2012 17:27: >> On 25 September 2012 16:12, Stefan Behnel wrote: >>> Stefan Behnel, 25.09.2012 14:55: >>>> there is a reference leak in the new dict iteration code in 0.17: >>>> >>>> http://trac.cython.org/cython_trac/ticket/790 >>>> >>>> I'll see if I can fix it and then release a pure bugfix 0.17.1 right >>>> afterwards. >>>> >>>> Please take a look at your own patch queues to see if there are any other >>>> important bugs that you would like to see fixed in that release. >>> >>> Update: I fixed this, so 0.17.1 will be released tomorrow (CEST) unless >>> you've thrown in your veto by then. >>> >>> @Mark: what about the NULL strides changes? Would you consider them safe >>> enough to go in or can they wait a bit? If unsure, please choose the latter. ;) >> >> Thanks for asking. Yeah I merged it last weekend with tests and error >> checking code, it should be fine. > > Ok - it won't impact existing code but fixes the implementation, so I think > it can go in. > > Could you merge it over? > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Done. We only have a few really small fixes, do we really want to push out a bugfix release with this? From stefan_ml at behnel.de Wed Sep 26 09:28:29 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 26 Sep 2012 09:28:29 +0200 Subject: [Cython] ref-leak in 0.17 In-Reply-To: References: <5061A9CD.4040508@behnel.de> <5061C9F1.5030106@behnel.de> <5061D24B.2010500@behnel.de> Message-ID: <5062AE9D.2000103@behnel.de> mark florisson, 25.09.2012 22:37: >>> On 25 September 2012 16:12, Stefan Behnel wrote: >>>> Stefan Behnel, 25.09.2012 14:55: >>>>> there is a reference leak in the new dict iteration code in 0.17: >>>>> >>>>> http://trac.cython.org/cython_trac/ticket/790 >>>>> >>>>> I'll see if I can fix it and then release a pure bugfix 0.17.1 right >>>>> afterwards. >>>>> >>>>> Please take a look at your own patch queues to see if there are any other >>>>> important bugs that you would like to see fixed in that release. >>>> >>>> Update: I fixed this, so 0.17.1 will be released tomorrow (CEST) unless >>>> you've thrown in your veto by then. > > We only have a few really small fixes, do we really want to push > out a bugfix release with this? My answer is Yes. The dict iteration regression breaks lxml really badly and I'm sure code of other people is impacted, too. Remember that ref-leaks are way less obvious than crashes and usually won't show in test suites. It took me quite a while to get to the source of this problem. Besides, it's been almost a month since the release and if nothing else piled up since then, the better. Stefan From robertwb at gmail.com Wed Sep 26 10:10:33 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 26 Sep 2012 01:10:33 -0700 Subject: [Cython] ref-leak in 0.17 In-Reply-To: <5062AE9D.2000103@behnel.de> References: <5061A9CD.4040508@behnel.de> <5061C9F1.5030106@behnel.de> <5061D24B.2010500@behnel.de> <5062AE9D.2000103@behnel.de> Message-ID: On Wed, Sep 26, 2012 at 12:28 AM, Stefan Behnel wrote: > mark florisson, 25.09.2012 22:37: >>>> On 25 September 2012 16:12, Stefan Behnel wrote: >>>>> Stefan Behnel, 25.09.2012 14:55: >>>>>> there is a reference leak in the new dict iteration code in 0.17: >>>>>> >>>>>> http://trac.cython.org/cython_trac/ticket/790 >>>>>> >>>>>> I'll see if I can fix it and then release a pure bugfix 0.17.1 right >>>>>> afterwards. >>>>>> >>>>>> Please take a look at your own patch queues to see if there are any other >>>>>> important bugs that you would like to see fixed in that release. >>>>> >>>>> Update: I fixed this, so 0.17.1 will be released tomorrow (CEST) unless >>>>> you've thrown in your veto by then. >> >> We only have a few really small fixes, do we really want to push >> out a bugfix release with this? > > My answer is Yes. The dict iteration regression breaks lxml really badly > and I'm sure code of other people is impacted, too. Remember that ref-leaks > are way less obvious than crashes and usually won't show in test suites. It > took me quite a while to get to the source of this problem. > > Besides, it's been almost a month since the release and if nothing else > piled up since then, the better. Also, on a more general note, I would rather err towards more small bugfix releases (when impactful bugs come up) instead of the pattern we've fallen into of only releasing a couple times a year. So +1 to this. - Robert From stefan_ml at behnel.de Wed Sep 26 10:44:29 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 26 Sep 2012 10:44:29 +0200 Subject: [Cython] [cython-users] cython error return values and pure python mode In-Reply-To: <34547d16-d2b7-40b1-bc3c-b082710a7bbd@googlegroups.com> References: <34547d16-d2b7-40b1-bc3c-b082710a7bbd@googlegroups.com> Message-ID: <5062C06D.70600@behnel.de> Luca Dall'Olio, 26.09.2012 10:24: > I just converted some cython code using the "@cython.returns(a_type)" > annotation and it works like a charm, thanks for that! :-) It was mostly an oversight that it hasn't been available for so long. The pure Python mode is still incomplete and gradually improving, mostly because the core developers do not use it much themselves and rarely notice what's missing. So it's always worth a) bringing up issues (as you did here, thanks!) and b) coming up with proposals and patches to make it more widely usable. We warmly welcome any contributions. > I was wondering how to convert in pure python mode the error return value > "except" clause ( > http://docs.cython.org/src/userguide/language_basics.html#error-return-values) > : > > cdef int spam() except -1: > ... > > @cython.cfunc > @cython.returns(cython.int) > def spam(): > ... > > Is there any way to specify that -1 is the error return value in pure > python mode ? No, not as for as I can tell. I think it would be worth adding that to the 'returns' annotation, e.g. @cython.returns(cython.int, on_error="-1") for "except -1". Similarly: @cython.returns(cython.int, on_error="-1", check=True) for "except? -1" and @cython.returns(cython.int, on_error="+") @cython.returns(cython.void, on_error="*") for "except +" and "except *" respectively. OTOH, maybe "except *" could actually be the default in pure Python mode? Stefan From stefan_ml at behnel.de Wed Sep 26 19:15:34 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 26 Sep 2012 19:15:34 +0200 Subject: [Cython] Cython 0.17.1 released Message-ID: <50633836.3020401@behnel.de> Hi all, I just released Cython 0.17.1. This is a bugfix-only release for the latest stable 0.17 release series. http://pypi.python.org/pypi/Cython/0.17.1 http://wiki.cython.org/ReleaseNotes-0.17.1 Two bugs were fixed in this release: * A reference leak was fixed in the new dict iteration code when the loop target was not a plain variable but an unpacked tuple. * Memory views did not handle the special case of a NULL buffer strides value, as allowed by PEP3118. Have fun, Stefan From stefan_ml at behnel.de Fri Sep 28 14:36:21 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 28 Sep 2012 14:36:21 +0200 Subject: [Cython] changelog for regular releases (was: ref-leak in 0.17) In-Reply-To: References: <5061A9CD.4040508@behnel.de> <5061C9F1.5030106@behnel.de> <5061D24B.2010500@behnel.de> <5062AE9D.2000103@behnel.de> Message-ID: <506599C5.1070600@behnel.de> Robert Bradshaw, 26.09.2012 10:10: > Also, on a more general note, I would rather err towards more small > bugfix releases (when impactful bugs come up) instead of the pattern > we've fallen into of only releasing a couple times a year. One thing I noticed for this tiny release was the overhead of creating a separate release notes wiki page for it to list all changes there. What about keeping the current changelog in the source tree, so that we can immediately add comments to it when we commit changes that we consider noteworthy? Stefan From lists at onerussian.com Sun Sep 30 01:08:38 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sat, 29 Sep 2012 19:08:38 -0400 Subject: [Cython] Cython 0.17 final released In-Reply-To: <5061BDB3.2020505@behnel.de> References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> <50430D67.5050608@behnel.de> <5061BDB3.2020505@behnel.de> Message-ID: <20120929230838.GK5955@onerussian.com> On Tue, 25 Sep 2012, Stefan Behnel wrote: > > Any chance we can tag the official 0.17 release in Github as well? > BTW, do we use lightweight tags or annotated tags in git? And why? you should use annotated -a (and/or signed -s, which are annotated) tags you do not use annotated tags: $> git describe upstream/master fatal: No annotated tags can describe 'd0cf23e7dadebdb31064ef8c1fddeaef049622dc'. However, there were unannotated tags: try --tags. $> git describe --tags upstream/master 0.17.1-26-gd0cf23e why should you use them? http://stackoverflow.com/questions/4971746/why-should-i-care-about-lightweight-vs-annotated-tags -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From lists at onerussian.com Sun Sep 30 01:09:56 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sat, 29 Sep 2012 19:09:56 -0400 Subject: [Cython] Cython 0.17.1 released In-Reply-To: <50633836.3020401@behnel.de> References: <50633836.3020401@behnel.de> Message-ID: <20120929230956.GL5955@onerussian.com> On Wed, 26 Sep 2012, Stefan Behnel wrote: > http://wiki.cython.org/ReleaseNotes-0.17.1 and a regular contribution: http://wiki.cython.org/ReleaseHistory references only 0.17 for me ;-) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From robertwb at gmail.com Sun Sep 30 06:31:53 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 29 Sep 2012 21:31:53 -0700 Subject: [Cython] Cython 0.17 final released In-Reply-To: <20120929230838.GK5955@onerussian.com> References: <5041B231.7020905@behnel.de> <50430175.1020104@behnel.de> <50430D67.5050608@behnel.de> <5061BDB3.2020505@behnel.de> <20120929230838.GK5955@onerussian.com> Message-ID: On Sat, Sep 29, 2012 at 4:08 PM, Yaroslav Halchenko wrote: > > On Tue, 25 Sep 2012, Stefan Behnel wrote: >> > Any chance we can tag the official 0.17 release in Github as well? >> BTW, do we use lightweight tags or annotated tags in git? And why? > > you should use annotated -a (and/or signed -s, which are annotated) tags > > you do not use annotated tags: > > $> git describe upstream/master > fatal: No annotated tags can describe 'd0cf23e7dadebdb31064ef8c1fddeaef049622dc'. > However, there were unannotated tags: try --tags. > > $> git describe --tags upstream/master > 0.17.1-26-gd0cf23e > > why should you use them? > > http://stackoverflow.com/questions/4971746/why-should-i-care-about-lightweight-vs-annotated-tags I haven't given it much thought before now, but you make a nice point for annotated tags. - Robert From robertwb at gmail.com Sun Sep 30 06:59:11 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 29 Sep 2012 21:59:11 -0700 Subject: [Cython] changelog for regular releases (was: ref-leak in 0.17) In-Reply-To: <506599C5.1070600@behnel.de> References: <5061A9CD.4040508@behnel.de> <5061C9F1.5030106@behnel.de> <5061D24B.2010500@behnel.de> <5062AE9D.2000103@behnel.de> <506599C5.1070600@behnel.de> Message-ID: On Fri, Sep 28, 2012 at 5:36 AM, Stefan Behnel wrote: > Robert Bradshaw, 26.09.2012 10:10: >> Also, on a more general note, I would rather err towards more small >> bugfix releases (when impactful bugs come up) instead of the pattern >> we've fallen into of only releasing a couple times a year. > > One thing I noticed for this tiny release was the overhead of creating a > separate release notes wiki page for it to list all changes there. What > about keeping the current changelog in the source tree, so that we can > immediately add comments to it when we commit changes that we consider > noteworthy? Sounds worthwhile to me; creating the wiki page after the fact is a pain (for large or small releases) and harder to remember to update when it's not in the repo. - Robert From stefan_ml at behnel.de Sun Sep 30 07:07:16 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 30 Sep 2012 07:07:16 +0200 Subject: [Cython] Cython 0.17.1 released In-Reply-To: <20120929230956.GL5955@onerussian.com> References: <50633836.3020401@behnel.de> <20120929230956.GL5955@onerussian.com> Message-ID: <5067D384.6030901@behnel.de> Yaroslav Halchenko, 30.09.2012 01:09: > > On Wed, 26 Sep 2012, Stefan Behnel wrote: >> http://wiki.cython.org/ReleaseNotes-0.17.1 > > and a regular contribution: > http://wiki.cython.org/ReleaseHistory > references only 0.17 for me > ;-) I added the link. Thanks. Stefan