From stefan_ml at behnel.de Sat Feb 12 06:33:32 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 12 Feb 2011 06:33:32 +0100 Subject: [Cython] Welcome to cython-devel@python.org Message-ID: <4D561BAC.509@behnel.de> Hi everyone, welcome to the new mailing list at python.org that replaces the original list cython-dev at codespeak.net. Stefan PS: this mail is mostly for testing the list and archive connections. From jyf1987 at gmail.com Sat Feb 12 06:37:34 2011 From: jyf1987 at gmail.com (Yunfan Jiang) Date: Sat, 12 Feb 2011 13:37:34 +0800 Subject: [Cython] Welcome to cython-devel@python.org In-Reply-To: <4D561BAC.509@behnel.de> References: <4D561BAC.509@behnel.de> Message-ID: is it means something? On Sat, Feb 12, 2011 at 1:33 PM, Stefan Behnel wrote: > Hi everyone, > > welcome to the new mailing list at python.org that replaces the original > list cython-dev at codespeak.net. > > Stefan > > PS: this mail is mostly for testing the list and archive connections. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -- ME = { ? ?"name": [ "jyf", "yunfan", "wuxian" ], ? ?"im": { ? ? ? ? "gtalk": "jyf1987 at gmail.com", ? ? ? ? "msn": "geek42 at live.cn" ? ? ? ? ? }, ? ?"job": "python engineer", ? ?"site": "http://hi.baidu.com/jyf1987", ? ?"interested": ?{ ? ? ? ?"tech": [ "linux", "python", "lua", "php", "html5", "c", "nosql"], ? ? ? ?"history": ["chinese history", "global history"], ? ? ? ?"SF": [ "hard SF", "Thought experiment" ], ? ? ? ?"music": [ "New Age", "Chinese old theme", "Electronic music", "Strange Music :}"] ? ? ?} ?} From monty at starfief.com Sat Feb 12 14:59:58 2011 From: monty at starfief.com (Rod Montgomery) Date: Sat, 12 Feb 2011 08:59:58 -0500 Subject: [Cython] Where is the separate cython-users mailing list? Message-ID: <4D56925E.2070308@starfief.com> Stefan Behnel wrote, on 11 Feb 2011 13:57:56 +0100: >...given that we have a separate cython-users mailing list anyway. Is there indeed a separate cython-users mailing list? If there is, would someone please tell me *where* it is? Thanks! From stefan_ml at behnel.de Sat Feb 12 15:22:13 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 12 Feb 2011 15:22:13 +0100 Subject: [Cython] Where is the separate cython-users mailing list? In-Reply-To: <4D56925E.2070308@starfief.com> References: <4D56925E.2070308@starfief.com> Message-ID: <4D569795.6000406@behnel.de> Rod Montgomery, 12.02.2011 14:59: > Stefan Behnel wrote, on 11 Feb 2011 13:57:56 +0100: > > >...given that we have a separate cython-users mailing list anyway. > > Is there indeed a separate cython-users mailing list? > > If there is, would someone please tell me *where* it is? I only added the "big fat link" from the Cython-dev list page a couple of months ago, so you may not have seen it at the time you subscribed. It's on Google Groups, so it's not surprising to me that it's Google's first hit for "cython-users". *wink* The list page is here: http://groups.google.com/group/cython-users In general, all Cython users are invited to join that list instead of cython-dev(el). It has a larger audience than cython-dev (including the core developers) and a much better focus on usage related questions. Stefan From sccolbert at gmail.com Sun Feb 13 21:09:39 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 13 Feb 2011 15:09:39 -0500 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 Message-ID: I have cython file which is using PyObject_GenericSetAttr which is defined as follows in my pyx file: cdef extern from "Python.h": int PyObject_GenericSetAttr(object, object, object) except -1 Now in my script I am using that function to generically delete an attribute by passing a NULL as the last value (this is proper way to trigger a generic delattr in the Python c-api) with the following line: PyObject_GenericSetAttr(self, name, NULL) # need to cast to object or Cython won't compile In Cython 0.14.1-1 this generates the following code: __pyx_t_3 = __pyx_v_self; __Pyx_INCREF(__pyx_t_3); __pyx_t_2 = __pyx_v_name; __Pyx_INCREF(__pyx_t_2); __pyx_t_4 = ((PyObject *)NULL); __Pyx_INCREF(__pyx_t_4); __pyx_t_5 = PyObject_GenericSetAttr(__pyx_t_3, __pyx_t_2, __pyx_t_4); if (unlikely(__pyx_t_5 == -1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 124; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0; __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0; __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0; This causes a segfault because the NULL is getting increfed via Py_INCREF instead of Py_XINCREF. This wasnt a problem in Cython 0.13-1 which generated the following code: __pyx_t_3 = PyObject_GenericSetAttr(__pyx_v_self, __pyx_v_name, ((PyObject *)NULL)); if (unlikely(__pyx_t_3 == -1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 124; __pyx_clineno = __LINE__; goto __pyx_L1_error;} and doesn't attempt to INCREF the NULL. Is there a workaround for my current situation? -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Feb 13 21:31:30 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 Feb 2011 09:31:30 +1300 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: References: Message-ID: <4D583FA2.4020900@canterbury.ac.nz> Chris Colbert wrote: > I have cython file which is using PyObject_GenericSetAttr > > Now in my script I am using that function to generically delete an > attribute by passing a NULL as the last value (this is proper way to > trigger a generic delattr in the Python c-api) I would have thought the proper way to do that was to use PyObject_DelAttr, which Pyrex exposes as delattr(). I don't think PyObject_GenericSetAttr is even meant to be called directly -- it's intended for filling the tp_setattr slot of type objects. > This causes a segfault because the NULL is getting increfed via > Py_INCREF instead of Py_XINCREF. I would recommend against trying to "fix" this. You got away with it before because you happened to be passing the NULL value directly to a function which is expecting it. But casting NULL to an object reference is not something that should be encouraged, because in any other context it would quickly lead to disaster. -- Greg From sccolbert at gmail.com Sun Feb 13 21:37:26 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 13 Feb 2011 15:37:26 -0500 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: <4D583FA2.4020900@canterbury.ac.nz> References: <4D583FA2.4020900@canterbury.ac.nz> Message-ID: The problem with delattr (and thus PyObject_DelAttr) arises when you define a __delattr__ method on your class. There is not easy way to then call back into the "normal" python delattr semantics, except by doing object.__delattr__ (which is not optimized by Cython). Futher, calling PyObject_GenericSetattr(obj, name, NULL) appears to be the proper use, given that it properly follows the descriptor chain and will call __delete__ if obj.name is a descriptor. I would argue that there should be at least some way to pass a NULL pointer in Cython where a PyObject* is expected. On Sun, Feb 13, 2011 at 3:31 PM, Greg Ewing wrote: > Chris Colbert wrote: > >> I have cython file which is using PyObject_GenericSetAttr >> Now in my script I am using that function to generically delete an >> attribute by passing a NULL as the last value (this is proper way to trigger >> a generic delattr in the Python c-api) >> > > I would have thought the proper way to do that was to use > PyObject_DelAttr, which Pyrex exposes as delattr(). > > I don't think PyObject_GenericSetAttr is even meant to be > called directly -- it's intended for filling the tp_setattr > slot of type objects. > > > This causes a segfault because the NULL is getting increfed via Py_INCREF >> instead of Py_XINCREF. >> > > I would recommend against trying to "fix" this. You got > away with it before because you happened to be passing the > NULL value directly to a function which is expecting it. > But casting NULL to an object reference is not something > that should be encouraged, because in any other context it > would quickly lead to disaster. > > -- > Greg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Sun Feb 13 21:39:39 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 13 Feb 2011 15:39:39 -0500 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: References: <4D583FA2.4020900@canterbury.ac.nz> Message-ID: I should mention, that these PyObject_Generic* functions are being used from with __getattribute__, __setattr__, and __delattr__ methods. I should have added that context in the begining. On Sun, Feb 13, 2011 at 3:37 PM, Chris Colbert wrote: > The problem with delattr (and thus PyObject_DelAttr) arises when you define > a __delattr__ method on your class. There is not easy way to then call back > into the "normal" python delattr semantics, except by doing > object.__delattr__ (which is not optimized by Cython). > > Futher, calling PyObject_GenericSetattr(obj, name, NULL) appears to be the > proper use, given that it properly follows the descriptor chain and will > call __delete__ if obj.name is a descriptor. > > I would argue that there should be at least some way to pass a NULL pointer > in Cython where a PyObject* is expected. > > > On Sun, Feb 13, 2011 at 3:31 PM, Greg Ewing wrote: > >> Chris Colbert wrote: >> >>> I have cython file which is using PyObject_GenericSetAttr >>> Now in my script I am using that function to generically delete an >>> attribute by passing a NULL as the last value (this is proper way to trigger >>> a generic delattr in the Python c-api) >>> >> >> I would have thought the proper way to do that was to use >> PyObject_DelAttr, which Pyrex exposes as delattr(). >> >> I don't think PyObject_GenericSetAttr is even meant to be >> called directly -- it's intended for filling the tp_setattr >> slot of type objects. >> >> >> This causes a segfault because the NULL is getting increfed via Py_INCREF >>> instead of Py_XINCREF. >>> >> >> I would recommend against trying to "fix" this. You got >> away with it before because you happened to be passing the >> NULL value directly to a function which is expecting it. >> But casting NULL to an object reference is not something >> that should be encouraged, because in any other context it >> would quickly lead to disaster. >> >> -- >> Greg >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Sun Feb 13 21:49:04 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Sun, 13 Feb 2011 15:49:04 -0500 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: References: <4D583FA2.4020900@canterbury.ac.nz> Message-ID: changing the include definition to: cdef extern from "Python.h": int PyObject_GenericSetAttr(PyObject*, PyObject*, PyObject*) except -1 Seems to solve the problem, with the inconvenience of needing to explicitly cast to before calling it. However, the objects are no longer incref'd before calling, so this may or not be an issue. On Sun, Feb 13, 2011 at 3:39 PM, Chris Colbert wrote: > I should mention, that these PyObject_Generic* functions are being used > from with __getattribute__, __setattr__, and __delattr__ methods. I should > have added that context in the begining. > > > On Sun, Feb 13, 2011 at 3:37 PM, Chris Colbert wrote: > >> The problem with delattr (and thus PyObject_DelAttr) arises when you >> define a __delattr__ method on your class. There is not easy way to then >> call back into the "normal" python delattr semantics, except by doing >> object.__delattr__ (which is not optimized by Cython). >> >> Futher, calling PyObject_GenericSetattr(obj, name, NULL) appears to be the >> proper use, given that it properly follows the descriptor chain and will >> call __delete__ if obj.name is a descriptor. >> >> I would argue that there should be at least some way to pass a NULL >> pointer in Cython where a PyObject* is expected. >> >> >> On Sun, Feb 13, 2011 at 3:31 PM, Greg Ewing wrote: >> >>> Chris Colbert wrote: >>> >>>> I have cython file which is using PyObject_GenericSetAttr >>>> Now in my script I am using that function to generically delete an >>>> attribute by passing a NULL as the last value (this is proper way to trigger >>>> a generic delattr in the Python c-api) >>>> >>> >>> I would have thought the proper way to do that was to use >>> PyObject_DelAttr, which Pyrex exposes as delattr(). >>> >>> I don't think PyObject_GenericSetAttr is even meant to be >>> called directly -- it's intended for filling the tp_setattr >>> slot of type objects. >>> >>> >>> This causes a segfault because the NULL is getting increfed via >>>> Py_INCREF instead of Py_XINCREF. >>>> >>> >>> I would recommend against trying to "fix" this. You got >>> away with it before because you happened to be passing the >>> NULL value directly to a function which is expecting it. >>> But casting NULL to an object reference is not something >>> that should be encouraged, because in any other context it >>> would quickly lead to disaster. >>> >>> -- >>> Greg >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Feb 14 07:49:42 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 Feb 2011 19:49:42 +1300 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: References: <4D583FA2.4020900@canterbury.ac.nz> Message-ID: <4D58D086.1000400@canterbury.ac.nz> Chris Colbert wrote: > The problem with delattr (and thus PyObject_DelAttr) arises when you > define a __delattr__ method on your class. There is not easy way to then > call back into the "normal" python delattr semantics, except by doing > object.__delattr__ (which is not optimized by Cython). Hmmm, perhaps it should be? And similarly for all the other type slots. > I would argue that there should be at least some way to pass a NULL > pointer in Cython where a PyObject* is expected. The trouble is that supporting this in general both safely and efficiently would require keeping track of which object references are allowed to be NULL. Could be done, but might require quite a lot of work. Maybe something more limited could be done such as allowing a literal NULL to be passed to a function argument that is specially marked as permitting this. -- Greg From greg.ewing at canterbury.ac.nz Mon Feb 14 07:50:03 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 14 Feb 2011 19:50:03 +1300 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: References: <4D583FA2.4020900@canterbury.ac.nz> Message-ID: <4D58D09B.2010708@canterbury.ac.nz> Chris Colbert wrote: > changing the include definition to: > > cdef extern from "Python.h": > int PyObject_GenericSetAttr(PyObject*, PyObject*, PyObject*) except -1 This suggests another possible workaround: cdef extern from "Python.h": int PyObject_GenericDelAttr "PyObject_GenericSetAttr" (object, object, PyObject*) This creates an alias for PyObject_GenericSetAttr with a different signature, which you then call as PyObject_GenericDelAttr(obj, name, NULL) If you're willing to write a tiny bit of C, you could make this tidier by using a .h file containing #define PyObject_GenericDelAttr(obj, name) PyObject_GenericSetAttr(obj, name, NULL) and then declare PyObject_GenericDelAttr to Cython with just two arguments. Maybe Cython should include something like this in its standard preamble, along with similar macros for other NULL-taking API functions. -- Greg From vitja.makarov at gmail.com Mon Feb 14 08:40:12 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 14 Feb 2011 10:40:12 +0300 Subject: [Cython] Control flow graph Message-ID: Hi! In order to implement "reaching definitions" algorithm. I'm now working on control-flow (or data-flow) graph. Here is funny picture made with graphviz ;) http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ -- vitja. From berthold.hoellmann at gl-group.com Mon Feb 14 09:08:13 2011 From: berthold.hoellmann at gl-group.com (Berthold Hoellmann) Date: Mon, 14 Feb 2011 09:08:13 +0100 Subject: [Cython] Welcome to cython-devel@python.org In-Reply-To: <4D561BAC.509@behnel.de> (Stefan Behnel's message of "Sat, 12 Feb2011 06:33:32 +0100") References: <4D561BAC.509@behnel.de> Message-ID: Stefan Behnel writes: > Hi everyone, > > welcome to the new mailing list at python.org that replaces the original > list cython-dev at codespeak.net. Will this list also be made avaliable via gmane as newsgroup "gmane.comp.python.cython.devel"? Kind regards Berthold H?llmann -- Germanischer Lloyd SE Berthold H?llmann Project Engineer, CAE Development Brooktorkai 18 20457 Hamburg Germany Phone: +49(0)40 36149-7374 Fax: +49(0)40 36149-7320 e-mail: berthold.hoellmann at gl-group.com Internet: http://www.gl-group.com This e-mail and any attachment thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. Any access of third parties to this e-mail is unauthorised. Any use of this e-mail by unintended recipients such as total or partial copying, distribution, disclosure etc. is prohibited and may be unlawful. When addressed to our clients the content of this e-mail is subject to the General Terms and Conditions of GL's Group of Companies applicable at the date of this e-mail. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. GL's Group of Companies does not warrant and/or guarantee that this message at the moment of receipt is authentic, correct and its communication free of errors, interruption etc. Germanischer Lloyd SE, 115442 AG HH, Hamburg, Vorstand: Erik van der Noordaa, Dr. Joachim Segatz, Pekka Paasivaara, Vorsitzender des Aufsichtsrats: Dr. Wolfgang Peiner From stefan_ml at behnel.de Mon Feb 14 09:17:57 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 14 Feb 2011 09:17:57 +0100 Subject: [Cython] Welcome to cython-devel@python.org In-Reply-To: References: <4D561BAC.509@behnel.de> Message-ID: <4D58E535.7020800@behnel.de> Berthold Hoellmann, 14.02.2011 09:08: > Stefan Behnel writes: >> welcome to the new mailing list at python.org that replaces the original >> list cython-dev at codespeak.net. > > Will this list also be made avaliable via gmane as newsgroup > "gmane.comp.python.cython.devel"? I've already requested an address change last week but didn't get a response over the weekend. It seems that it's not enough to copy over their subscription address. Maybe they ignore e-mails to that address that are not directed at the specific mailing list address they know. I hope it'll get fixed up soon. Stefan From stefan_ml at behnel.de Mon Feb 14 09:31:13 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 14 Feb 2011 09:31:13 +0100 Subject: [Cython] Control flow graph In-Reply-To: References: Message-ID: <4D58E851.7010804@behnel.de> Vitja Makarov, 14.02.2011 08:40: > In order to implement "reaching definitions" algorithm. > I'm now working on control-flow (or data-flow) graph. Cool. Another good topic for the workshop. I've added it to the list on the wiki page. Note that it's related to the NameNode graph, but not completely overlapping. The latter may be considered an application of a more general control flow graph. > Here is funny picture made with graphviz ;) > > http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ Nice one. One thing it's missing is the implicit return at the end of a function body. This is something that's needed in order to figure out what the return value of a function is. It would also make the graphs more readable, as you'd actually see what happens after a statement with no outbound arrows. Stefan From dagss at student.matnat.uio.no Mon Feb 14 11:48:35 2011 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 14 Feb 2011 11:48:35 +0100 Subject: [Cython] Control flow graph In-Reply-To: References: Message-ID: <4D590883.5030500@student.matnat.uio.no> On 02/14/2011 08:40 AM, Vitja Makarov wrote: > Hi! > > In order to implement "reaching definitions" algorithm. > I'm now working on control-flow (or data-flow) graph. > > Here is funny picture made with graphviz ;) > > http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ > > Cool! This will be useful for so much more. Dag Sverre From sccolbert at gmail.com Mon Feb 14 15:11:21 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 14 Feb 2011 09:11:21 -0500 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: <4D58D09B.2010708@canterbury.ac.nz> References: <4D583FA2.4020900@canterbury.ac.nz> <4D58D09B.2010708@canterbury.ac.nz> Message-ID: On Mon, Feb 14, 2011 at 1:50 AM, Greg Ewing wrote: > Chris Colbert wrote: > >> changing the include definition to: >> >> cdef extern from "Python.h": >> int PyObject_GenericSetAttr(PyObject*, PyObject*, PyObject*) except -1 >> > > This suggests another possible workaround: > > > cdef extern from "Python.h": > int PyObject_GenericDelAttr "PyObject_GenericSetAttr" (object, object, > PyObject*) > > This creates an alias for PyObject_GenericSetAttr with a different > signature, which you then call as > > Awesome, I didn't know you could do this. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Mon Feb 14 15:13:51 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 14 Feb 2011 09:13:51 -0500 Subject: [Cython] Bug in NULL handling introduced 0.14.1-1 In-Reply-To: <4D58D086.1000400@canterbury.ac.nz> References: <4D583FA2.4020900@canterbury.ac.nz> <4D58D086.1000400@canterbury.ac.nz> Message-ID: On Mon, Feb 14, 2011 at 1:49 AM, Greg Ewing wrote: > Chris Colbert wrote: > >> The problem with delattr (and thus PyObject_DelAttr) arises when you >> define a __delattr__ method on your class. There is not easy way to then >> call back into the "normal" python delattr semantics, except by doing >> object.__delattr__ (which is not optimized by Cython). >> > > Hmmm, perhaps it should be? And similarly for all the other type > slots. > > > I would argue that there should be at least some way to pass a NULL >> pointer in Cython where a PyObject* is expected. >> > > The trouble is that supporting this in general both safely and > efficiently would require keeping track of which object references > are allowed to be NULL. Could be done, but might require quite a > lot of work. > > Maybe something more limited could be done such as allowing > a literal NULL to be passed to a function argument that is > specially marked as permitting this. > > I had the same kind of thoughts. Perhaps something like: cdef extern from "Python.h": PyObject_GenericSetAttr(object, object, object or NULL) -------------- next part -------------- An HTML attachment was scrubbed... URL: From strombrg at gmail.com Mon Feb 14 20:09:00 2011 From: strombrg at gmail.com (Dan Stromberg) Date: Mon, 14 Feb 2011 11:09:00 -0800 Subject: [Cython] yield... In-Reply-To: References: Message-ID: On Sun, Feb 13, 2011 at 9:55 PM, Vitja Makarov wrote: > 2011/2/14 Dan Stromberg : > > > > On Sun, Feb 13, 2011 at 5:17 PM, Dan Stromberg > wrote: > >> > >> On Sun, Feb 6, 2011 at 3:38 AM, Vitja Makarov > >> wrote: > >>> > >>> 2011/2/6 Dan Stromberg : > >>> > > >>> > On Sat, Feb 5, 2011 at 3:10 PM, Vitja Makarov < > vitja.makarov at gmail.com> > >>> > wrote: > >>> >> > >>> >> 2011/2/6 Vitja Makarov : > >>> >> > 2011/2/6 Dan Stromberg : > >>> >> >> > >>> >> >> Just a quick note to let you know I'm very eager to have working > >>> >> >> generators > >>> >> >> (yield) in Cython... ^_^ It's frequently the main thing that > >>> >> >> complicates > >>> >> >> moving my critical sections to Cython. > >>> >> >> -- > >>> >> >> Dan Stromberg > >>> >> >> > >>> >> > > >>> >> > Ok, you will see generators soon in cython))) the most important > >>> >> > part > >>> >> > is already implemented, but I'm not sure that's the importetant > >>> >> > one;) > >>> >> > > >>> >> > >>> >> now you can use my github repo at https://github.com/vitek/cython > >>> > > >>> > Hey, that's great. > >>> > I tried a little test program with it, and it worked great. I'll > >>> > likely > >>> > give it a try with a more real-world workload before long. > >>> > Does it pass all the tests you've thrown at it yet? > >>> > Thanks! > >>> > > >>> > >>> Hmm now it work as excepted. If you will find a case when it doesn't > >>> work, report it. > >>> See also tests at tests/run/generators.pyx > >>> > >>> -- > >>> vitja. > >> > >> Do you want e-mail, or a formal bug tracking entry? > >> In case you want e-mail: > >> + /usr/local/cpython-2.5/bin/python /usr/local/cpython-2.5/bin/cython > >> rolling_checksum_pyx_mod.pyx > >> Error compiling Cython file: > >> ------------------------------------------------------------ > >> ... > >> import os > >> import sys > >> #mport comma_mod > >> def min_max_chunker(file_handle): > >> ^ > >> ------------------------------------------------------------ > >> rolling_checksum_pyx_mod.pyx:9:0: Compiler crash in MarkClosureVisitor > >> ModuleNode.body = StatListNode(rolling_checksum_pyx_mod.pyx:4:0) > >> StatListNode.stats[2] = StatListNode(rolling_checksum_pyx_mod.pyx:9:0) > >> StatListNode.stats[0] = DefNode(rolling_checksum_pyx_mod.pyx:9:0, > >> doc = u'Make sure chunk sizes are above and below a pair of > >> thresholds', > >> modifiers = [...]/0, > >> name = u'min_max_chunker', > >> num_required_args = 1, > >> reqd_kw_flags_cname = '0') > >> Compiler crash traceback from this point on: > >> File "Visitor.py", line 173, in > >> Cython.Compiler.Visitor.TreeVisitor._visitchild > >> (Cython/Compiler/Visitor.c:3508) > >> File > >> > "/usr/local/cpython-2.5/lib/python2.5/site-packages/Cython/Compiler/ParseTreeTransforms.py", > >> line 1380, in visit_FuncDefNode > >> for i, yield_expr in enumerate(collector.yields, 1): > >> TypeError: enumerate() takes exactly 1 argument (2 given) > >> make: *** [rcms] Error 1 > >> > >> > >> The code that produced this is > >> at http://stromberg.dnsalias.org/svn/backshift/trunk/ - with make > target > >> "rcms". It expects to find a variety of cpython's at > >> /usr/local/cpython-x.y/bin/python . > > > > > > Oh, and of course, to eliminate some details for you, here's the > > (automatically generated via m4) file that's giving the compiler error > > (attached). > > I suspect this issue may just be because a 2.x-ism (x>5) with how > enumerate > > is used has crept into cython. > > Thanks! > > > Hi! > > It seems that you have old python version, 2.4? > It's 2.5. > Here is a fix: > > https://github.com/vitek/cython/commit/bb8cd8275bf910117750356845814d4d9d998d6e Thanks, but... How do I apply this? I tried git pull and git clone, but neither worked for me. I know SVN pretty well, but I haven't used git much yet. -- Dan Stromberg -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Feb 14 23:04:20 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 Feb 2011 11:04:20 +1300 Subject: [Cython] Control flow graph In-Reply-To: References: Message-ID: <4D59A6E4.2090801@canterbury.ac.nz> Vitja Makarov wrote: > > Here is funny picture made with graphviz ;) > > http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ Gives the term "spaghetti code" a whole new meaning! -- Greg From vitja.makarov at gmail.com Tue Feb 15 06:50:51 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 15 Feb 2011 08:50:51 +0300 Subject: [Cython] Control flow graph In-Reply-To: References: Message-ID: 2011/2/15 Vitja Makarov : > 2011/2/15 Robert Bradshaw : >> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote: >>> Hi! >>> >>> In order to implement "reaching definitions" algorithm. >>> I'm now working on control-flow (or data-flow) graph. >>> >>> Here is funny picture made with graphviz ;) >>> >>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >> >> Cool. Any plans on handling exceptions? >> > Ooops forget to CC cython-devel... Sure, but I don't have much time for this :( Linear block inside try...except body should be split by assignments and each subblock should point to exception handling entry point. As result I want to have set of possible assignments for each NameNode position. So handling of uninitialized variables, unused, unused results should be easy, later it may help to implement local variable deletion. And guess that could help type inference. -- vitja. From robertwb at math.washington.edu Tue Feb 15 08:21:40 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 14 Feb 2011 23:21:40 -0800 Subject: [Cython] Control flow graph In-Reply-To: References: Message-ID: On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote: > 2011/2/15 Robert Bradshaw : >> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote: >>> Hi! >>> >>> In order to implement "reaching definitions" algorithm. >>> I'm now working on control-flow (or data-flow) graph. >>> >>> Here is funny picture made with graphviz ;) >>> >>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >> >> Cool. Any plans on handling exceptions? >> > > Sure, but I don't have much time for this :( > > Linear block inside try...except body should be split by assignments > and each subblock should point to exception handling entry point. Would every possible failing sub-expression have to point to the exception handling point(s)? I suppose it depends on whether you'll be handling more than assignment tracking. > As result I want to have set of possible assignments for each NameNode position. > > So handling of uninitialized variables, unused, unused results should be easy, > later it may help to implement local variable deletion. And guess that > could help type inference. Ye. - Robert From stefan_ml at behnel.de Tue Feb 15 08:38:22 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 Feb 2011 08:38:22 +0100 Subject: [Cython] Control flow graph In-Reply-To: References: Message-ID: <4D5A2D6E.9060406@behnel.de> Robert Bradshaw, 15.02.2011 08:21: > On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote: >> 2011/2/15 Robert Bradshaw: >>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote: >>>> Hi! >>>> >>>> In order to implement "reaching definitions" algorithm. >>>> I'm now working on control-flow (or data-flow) graph. >>>> >>>> Here is funny picture made with graphviz ;) >>>> >>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >>> >>> Cool. Any plans on handling exceptions? >> >> Sure, but I don't have much time for this :( >> >> Linear block inside try...except body should be split by assignments >> and each subblock should point to exception handling entry point. > > Would every possible failing sub-expression have to point to the > exception handling point(s)? Well, in most cases (especially the interesting ones), this will be the function exit point, so it'll be easy. And in some cases, we may be able to infer that a specific exception that an expression (e.g. arithmetics or a 'raise' statement) can raise will not get caught by a given except clause (although that's certainly a tricky optimisation). But in general, I think any subexpression that potentially raises an exception must point to the next exception handling point. > I suppose it depends on whether you'll be handling more than assignment > tracking. We *may* get away with a statement-level graph in that case, but I somehow doubt it already. For example, list comprehensions leak their variable in Py2 code, so it's important to know if they are executed or not, and they may appear in any kind of expression. Stefan From dagss at student.matnat.uio.no Tue Feb 15 09:49:02 2011 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 15 Feb 2011 09:49:02 +0100 Subject: [Cython] Control flow graph In-Reply-To: References: Message-ID: <4D5A3DFE.3090908@student.matnat.uio.no> On 02/15/2011 08:21 AM, Robert Bradshaw wrote: > On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote: > >> 2011/2/15 Robert Bradshaw: >> >>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote: >>> >>>> Hi! >>>> >>>> In order to implement "reaching definitions" algorithm. >>>> I'm now working on control-flow (or data-flow) graph. >>>> >>>> Here is funny picture made with graphviz ;) >>>> >>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >>>> >>> Cool. Any plans on handling exceptions? >>> >>> >> Sure, but I don't have much time for this :( >> >> Linear block inside try...except body should be split by assignments >> and each subblock should point to exception handling entry point. >> > Would every possible failing sub-expression have to point to the > exception handling point(s)? I suppose it depends on whether you'll be > handling more than assignment tracking. > > >> As result I want to have set of possible assignments for each NameNode position. >> >> So handling of uninitialized variables, unused, unused results should be easy, >> later it may help to implement local variable deletion. And guess that >> could help type inference. >> > Ye. > I'm thinking that transforming NameNode + various assignment/lookup nodes into "opcode nodes" such as SetLocalNode, GetLocalNode, SetGlobalNode, GetAttributeNode and so on would be a natural step to make such code cleaner. Then (after an isolated transform doing this) the logic would just need to act on individual nodes, not combination of nodes. Just an idea. Dag Sverre From vitja.makarov at gmail.com Tue Feb 15 10:59:31 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 15 Feb 2011 12:59:31 +0300 Subject: [Cython] Control flow graph In-Reply-To: <4D5A2D6E.9060406@behnel.de> References: <4D5A2D6E.9060406@behnel.de> Message-ID: 2011/2/15 Stefan Behnel : > Robert Bradshaw, 15.02.2011 08:21: >> >> On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote: >>> >>> 2011/2/15 Robert Bradshaw: >>>> >>>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote: >>>>> >>>>> Hi! >>>>> >>>>> In order to implement "reaching definitions" algorithm. >>>>> I'm now working on control-flow (or data-flow) graph. >>>>> >>>>> Here is funny picture made with graphviz ;) >>>>> >>>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >>>> >>>> Cool. Any plans on handling exceptions? >>> >>> Sure, but I don't have much time for this :( >>> >>> Linear block inside try...except body should be split by assignments >>> and each subblock should point to exception handling entry point. >> >> Would every possible failing sub-expression have to point to the >> exception handling point(s)? > Not sure here as now graph node is list of NameNode ref and AssignmentNode. I'm not sure but it seems that every sub-expression could raise exception now, is there any flag? > Well, in most cases (especially the interesting ones), this will be the > function exit point, so it'll be easy. And in some cases, we may be able to > infer that a specific exception that an expression (e.g. arithmetics or a > 'raise' statement) can raise will not get caught by a given except clause > (although that's certainly a tricky optimisation). > > But in general, I think any subexpression that potentially raises an > exception must point to the next exception handling point. > Right, and each exception handling point should have reference to the next one or function exit point. > >> I suppose it depends on whether you'll be handling more than assignment >> tracking. > > We *may* get away with a statement-level graph in that case, but I somehow > doubt it already. For example, list comprehensions leak their variable in > Py2 code, so it's important to know if they are executed or not, and they > may appear in any kind of expression. > This should be special case for scoped expression node. Now I build graph "from scratch" it include only name node references and assignments, and positions marks to draw gv. May be node tree should be translated into graph and then local variables graph could be created from it. On the other hand CreateAbstractGraph transformation could be used to create specialized graph. -- vitja. From stefan_ml at behnel.de Tue Feb 15 20:12:21 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 Feb 2011 20:12:21 +0100 Subject: [Cython] Cython builds on various Debian platforms In-Reply-To: <20110215185057.GY21658@onerussian.com> References: <4D4FB57C.3010600@behnel.de> <20110213144726.GS21658@onerussian.com> <4D57F1FE.90603@behnel.de> <4D57F4D7.2070008@behnel.de> <20110214203850.GM21658@onerussian.com> <4D5ABDE2.60004@behnel.de> <4D5AC06A.30107@behnel.de> <20110215185057.GY21658@onerussian.com> Message-ID: <4D5AD015.7040009@behnel.de> [CC-ing cython-devel here] Yaroslav Halchenko is Cython's current maintainer for Debian. Yaroslav Halchenko, 15.02.2011 19:50: >> Stefan Behnel, 15.02.2011 18:54: >>> Yaroslav Halchenko, 14.02.2011 21:38: >>>> I guess it might bring us >>>> surprises while building on those exotics platforms Debian supports ;) >>> >>> I don't really hope for major surprises, but it sure would be great to >>> support those platforms. >> >> BTW, I assume the build results are publicly available somewhere? > > SURE! > > as for packages: > ftp://ftp.debian.org/debian/pool/main/c/cython/ > > but more interesting -- logs > http://packages.qa.debian.org/c/cython.html > > is the top page for 'Debian developer information' on cython (you > might like to subscribe to the package tracking -- lower left corner). > > for builds across platforms go to the link "buildd: logs": > https://buildd.debian.org/pkg.cgi?pkg=cython > > so 0.14.1-1 was already built ok across platforms seems to me ;) Cool. That looks good. As you mentioned, it doesn't include the test runs yet, but it would be really great if you could get those running. I am a bit surprised that each platform compiles the Cython modules several times according to the logs. Is that intended? Stefan From vitja.makarov at gmail.com Tue Feb 15 22:37:25 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 16 Feb 2011 00:37:25 +0300 Subject: [Cython] Control flow graph In-Reply-To: <4D5A2D6E.9060406@behnel.de> References: <4D5A2D6E.9060406@behnel.de> Message-ID: 2011/2/15 Stefan Behnel : > Robert Bradshaw, 15.02.2011 08:21: >> >> On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote: >>> >>> 2011/2/15 Robert Bradshaw: >>>> >>>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote: >>>>> >>>>> Hi! >>>>> >>>>> In order to implement "reaching definitions" algorithm. >>>>> I'm now working on control-flow (or data-flow) graph. >>>>> >>>>> Here is funny picture made with graphviz ;) >>>>> >>>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >>>> >>>> Cool. Any plans on handling exceptions? >>> >>> Sure, but I don't have much time for this :( >>> >>> Linear block inside try...except body should be split by assignments >>> and each subblock should point to exception handling entry point. >> >> Would every possible failing sub-expression have to point to the >> exception handling point(s)? > > Well, in most cases (especially the interesting ones), this will be the > function exit point, so it'll be easy. And in some cases, we may be able to > infer that a specific exception that an expression (e.g. arithmetics or a > 'raise' statement) can raise will not get caught by a given except clause > (although that's certainly a tricky optimisation). > > But in general, I think any subexpression that potentially raises an > exception must point to the next exception handling point. > > >> I suppose it depends on whether you'll be handling more than assignment >> tracking. > > We *may* get away with a statement-level graph in that case, but I somehow > doubt it already. For example, list comprehensions leak their variable in > Py2 code, so it's important to know if they are executed or not, and they > may appear in any kind of expression. > Hmm... both python and codespeaks in the thread Here is my commit it's mostly broken now but anyway https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a -- vitja. From stefan_ml at behnel.de Wed Feb 16 07:09:50 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Feb 2011 07:09:50 +0100 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms Message-ID: <4D5B6A2E.8020303@behnel.de> [forwarding to the list, from Yaroslav Halchenko] Hi, I am a member of Python Applications Packaging Team in Debian. So now me, later some other members could chime in to provide fresh uploads. On Tue, 15 Feb 2011, Stefan Behnel wrote: > Cool. That looks good. As you mentioned, it doesn't include the test > runs yet, but it would be really great if you could get those > running. great to hear to be on the same page ;) the only additional concern I see is the duration -- it might torture build boxes for old/exotic architectures quite a bit... there are no timeouts (like with ctest platform -- got bitten by them a few times already) among your tests, aren't there? (just want to make sure) > I am a bit surprised that each platform compiles the Cython modules > several times according to the logs. Is that intended? should build once per each supported python version (currently 2.5 and 2.6) and for debug builds of those supported versions... looking into the log, indeed building seems to be invoked twice though -- I guess my fault with build/cython.1: build where build is a PHONY. I will look into it for the next upload: $> wget -O- -q https://buildd.debian.org/fetch.cgi\?pkg\=cython\;ver\=0.14.1-1\;arch\=hurd-i386\;stamp\=1297731063 | grep -B2 -A2 setup.py for buildver in 2.5 2.6; do \ cd /build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1 && cd . && \ python$buildver setup.py clean \ -a; \ done -- set -e; for buildver in 2.5 2.6; do \ cd /build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1 && cd . && \ python$buildver setup.py build \ --build-base="/build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1/./build"; \ done -- set -e; for buildver in 2.5 2.6; do \ cd /build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1 && cd . && \ python$buildver setup.py build \ --build-base="/build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1/./build"; \ done -- set -e; for buildver in 2.5 2.6; do \ cd /build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1 && cd . && \ python$buildver setup.py install \ --root="/build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1/debian/cython" \ --install-purelib=/usr/lib/python$buildver/site-packages/ \ -- dh_installdirs -pcython-dbg for i in 2.5 2.6; do \ python$i-dbg ./setup.py install --prefix=/usr --no-compile -O0 \ --root /build/buildd-cython_0.14.1-1-hurd-i386-fJZAFo/cython-0.14.1/debian/cython-dbg; \ done -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic From stefan_ml at behnel.de Wed Feb 16 07:11:32 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Feb 2011 07:11:32 +0100 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms Message-ID: <4D5B6A94.6050707@behnel.de> [forwarding to the list, from Yaroslav Halchenko] On Tue, 15 Feb 2011, Stefan Behnel wrote: > Cool. That looks good. As you mentioned, it doesn't include the test > runs yet, but it would be really great if you could get those > running. ok - added testing during build, so now need to assure that everything passes and all necessary dependencies are there. So, there is that failing one I have mentioned before: ,--- | ====================================================================== | FAIL: test_embed (__main__.EmbedTest) | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "runtests.py", line 886, in test_embed | "make PYTHON='%s' LIBDIR1='%s' test > make.output" % (sys.executable, libdir)) == 0) | AssertionError `--- what could be done about it or should it be excluded? -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic From matthew at kerkhofftech.ca Wed Feb 16 20:22:16 2011 From: matthew at kerkhofftech.ca (Matthew Fox) Date: Wed, 16 Feb 2011 19:22:16 +0000 Subject: [Cython] Possible Freeze-related regression from 0.14 to 0.14.1 Message-ID: <136CE5D04DC5244997EB0603E22110E5083655B2@kt-exchange1.kerkhofftech.local> My project using Cython-freeze compiles properly with Cython 0.14.0 but fails with 0.14.1. I run Python 2.5 on Debian Lenny and the cython_freeze from github (October 29, 2009). The attached simple test case (1 plain .pyx and .pxd, 1 .pyx with main(), and a makefile) demonstrates the issue- it works under 0.14.0 and fails under 0.14.1. The issue appears to be that Cython 0.14.0 declares an "int __pyx_module_is_main_mainster = 0" in mainster.c, while 0.14.1 declares it as static: "static int __pyx_module_is_main_ mainster = 0". When linking the object files, this results in the error from GCC: gcc -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions -L/usr/lib/python2.5/config executable.o mainster.o foo.o -lpython2.5 -lm -lpthread -ldl -lutil -o executable executable.o: In function `main': [...]/executable.c:54: undefined reference to `__pyx_module_is_main_mainster' collect2: ld returned 1 exit status make: *** [executable] Error 1 If I remove "static" from the declaration in mainster.c, I can recompile it and link the object files as I could do with Cython 0.14.0. Let me know if you need more details or if it's not a bug at all and there's a workaround. My workaround for now will be to use 0.14.0. Thanks, Matt Fox Phone: 604-824-2770 x305 Toll free: 800-360-2319 Email: matthew at kerkhofftech.ca Web: http://www.kerkhofftech.ca From matthew at kerkhofftech.ca Wed Feb 16 20:23:40 2011 From: matthew at kerkhofftech.ca (Matthew Fox) Date: Wed, 16 Feb 2011 19:23:40 +0000 Subject: [Cython] Possible Freeze-related regression from 0.14 to 0.14.1 Message-ID: <136CE5D04DC5244997EB0603E22110E5083655CB@kt-exchange1.kerkhofftech.local> Of course I would forget to attach the test case. Let's try again. > -----Original Message----- > From: Matthew Fox > Sent: February-16-11 11:22 AM > To: 'cython-devel at python.org' > Subject: Possible Freeze-related regression from 0.14 to 0.14.1 > > My project using Cython-freeze compiles properly with Cython 0.14.0 but > fails with 0.14.1. I run Python 2.5 on Debian Lenny and the > cython_freeze from github (October 29, 2009). The attached simple test > case (1 plain .pyx and .pxd, 1 .pyx with main(), and a makefile) > demonstrates the issue- it works under 0.14.0 and fails under 0.14.1. > > The issue appears to be that Cython 0.14.0 declares an "int > __pyx_module_is_main_mainster = 0" in mainster.c, while 0.14.1 declares > it as static: "static int __pyx_module_is_main_ mainster = 0". When > linking the object files, this results in the error from GCC: > > gcc -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions - > L/usr/lib/python2.5/config executable.o mainster.o foo.o -lpython2.5 > -lm -lpthread -ldl -lutil -o executable > executable.o: In function `main': > [...]/executable.c:54: undefined reference to > `__pyx_module_is_main_mainster' > collect2: ld returned 1 exit status > make: *** [executable] Error 1 > > If I remove "static" from the declaration in mainster.c, I can > recompile it and link the object files as I could do with Cython > 0.14.0. > > Let me know if you need more details or if it's not a bug at all and > there's a workaround. My workaround for now will be to use 0.14.0. > > Thanks, > Matt Fox > > > > Phone: 604-824-2770 x305 > Toll free: 800-360-2319 > Email: matthew at kerkhofftech.ca > Web: http://www.kerkhofftech.ca -------------- next part -------------- A non-text attachment was scrubbed... Name: freezebug.tar.gz Type: application/x-gzip Size: 1254 bytes Desc: freezebug.tar.gz URL: From dalcinl at gmail.com Thu Feb 17 00:17:12 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 16 Feb 2011 20:17:12 -0300 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: <4D5B6A94.6050707@behnel.de> References: <4D5B6A94.6050707@behnel.de> Message-ID: On 16 February 2011 03:11, Stefan Behnel wrote: > [forwarding to the list, from Yaroslav Halchenko] > > On Tue, 15 Feb 2011, Stefan Behnel wrote: >> >> Cool. That looks good. As you mentioned, it doesn't include the test >> runs yet, but it would be really great if you could get those >> running. > > ok - added testing during build, so now need to assure that everything > passes and all necessary dependencies are there. ?So, there is that > failing one I have mentioned before: > > ,--- > | ====================================================================== > | FAIL: test_embed (__main__.EmbedTest) > | ---------------------------------------------------------------------- > | Traceback (most recent call last): > | ? File "runtests.py", line 886, in test_embed > | ? ? "make PYTHON='%s' LIBDIR1='%s' test > make.output" % (sys.executable, > libdir)) == 0) > | AssertionError > `--- > > what could be done about it or should it be excluded? > I've pushed some fixes. Now this testcase should run from ancient Python 2.3 to head Python 3.2, both for static and sharedlib builds (but not in Windows). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From jason-sage at creativetrax.com Thu Feb 17 09:34:55 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Thu, 17 Feb 2011 00:34:55 -0800 Subject: [Cython] documentation tarball Message-ID: <4D5CDDAF.5020406@creativetrax.com> Would it be too much trouble to post a snapshot of the cython-docs tree along with a cython release? I see the git repository for cython-docs, but it would be more convenient for an end-user who wants a copy of the docs to not have to search through the commit log to find the right commit corresponding to a release. For example, it would be great if a zip or tarball of the documentation for 0.14.1 was posted on the cython.org homepage in the Download section, perhaps with text like this: The latest release of Cython is 0.14.1 (released 2011-02-04). You can download it as a gzipped tar or as a zip file. The documentation is also available (gzipped tar or zip). Thanks, Jason From lists at onerussian.com Thu Feb 17 13:38:40 2011 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 17 Feb 2011 07:38:40 -0500 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: References: <4D5B6A94.6050707@behnel.de> Message-ID: <20110217123840.GF21658@onerussian.com> On Wed, 16 Feb 2011, Lisandro Dalcin wrote: > > | AssertionError > > `--- > > what could be done about it or should it be excluded? > I've pushed some fixes. Now this testcase should run from ancient > Python 2.3 to head Python 3.2, both for static and sharedlib builds > (but not in Windows). first of all THANKS for the patch -- I picked it up into 0.14.1-2 debian package. Now tests are enabled and I just uploaded 0.14.1-2 into Debian -- lets see how it would go across architectures ;-) 2nd -- THANKS for ...: at first I got confused why I saw no commits since Dec in HG clone of Cython I had, and then mentioned that you moved to using GIT and github. Awesome and thank you for taking care about my sanity (although unintentionally I guess ;) ) -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic From wking at drexel.edu Thu Feb 17 14:51:13 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 17 Feb 2011 08:51:13 -0500 Subject: [Cython] Outdated `hg export` on cython-devel homepage Message-ID: <20110217135113.GB9353@tyr.home.net> The `hg export` on the cython-devel page [1] should probably be changed to (or additionally list) `git format-patch`, now that Cython's versioned in Git. Keeping the `hg export` reference might be useful for Mercurial lovers using hg-git. [1]: http://mail.python.org/mailman/listinfo/cython-devel -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From wking at drexel.edu Thu Feb 17 14:29:41 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 17 Feb 2011 08:29:41 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> Message-ID: <20110217132940.GA28686@tyr.home.net> This thread is coming over to cython-dev (and the new cython-devel) from cython-users because it turns out it will probably require chaning the Cython code. To get everyone who hasn't been following on cython-users up to speed, here's a summary of what I'm trying to do: That's what I was trying to give with this: On Wed, Feb 09, 2011 at 12:23:25PM -0500, W. Trevor King wrote: > I'm wrapping an external C library with Cython, so I have `mylib.pxd`: > > cdef extern from 'mylib.h' > enum: CONST_A > enum: CONST_B > ... > > where I declare each constant macro from the library's header `mylib.h`: > > #define CONST_A 1 > #define CONST_B 2 > ... > > Now I want to expose those constants in Python, so I have `expose.pyx`: > > cimport mylib > > CONST_A = mylib.CONST_A > CONST_B = mylib.CONST_B > ... > > But the last part seems pretty silly. I'd like to do something like > > cimport mylib > import sys > > for name in dir(mylib): > setattr(sys.modules[__name__], name, getattr(mylib, name)) > > which compiles fine, but fails to import with... Looking into the Cython internals, everything defined in mylib.pxd is stored as `Entry`s in a `ModuleScope`, and... On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: > On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: > > What I'm missing is a way to bind the ModuleScope namespace to a name > > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, > > name)` will work in expose.pyx. > > You have also hit into the thorny issue that .pxd files are used for > many things. They may be pure C library declarations with no Python > module backing, they may be declarations of (externally implemented) > Python modules (such as numpy.pxd), or they may be declarations for > Cython-implemented modules. > > It seems like it would be easier to generate some kind of wrapper > > class (PxdModule?) for mylib when it is cimported (at compile time), > > and then further interactions would take care of themselves (at run > > time). > > Would such an object be created anew for every module that cimports > the declaration file? Hmm, That doesn't sound very nice, does it. However, .pxd files declaring C libraries have no Python-space presence, so that was my initial idea. > I have toyed with the idea of subclassing the module object itself for > better support of C-level attributes from the Python (and Cython) > namespaces. Sorry, I don't understand "better support of C-level attributes". Can you give an example? > Here's another idea, what if extern blocks could contain cpdef > declarations, which would automatically generate a Python-level > wrappers for the declared members (if possible, otherwise an error)? Ah, this sounds good! Of the three .pxd roles you list above, external Python modules (e.g. numpy) and Cython-implemented modules (e.g. matched .pxd/.pyx) both already have a presence in Python-space. What's missing is a way to give (where possible) declarations of external C libraries a Python presence. cpdef fills this hole nicely, since its whole purpose is to expose Python interfaces to C-based elements. A side effect of this cpdef change would be that now even bare .pxd files (no matching .pyx) would have a Python presence, so You could do something like cimport mylib as mylib_c import mylib as mylib_py import sys # Access through Python for name in dir(mylib_py): setattr(sys.modules[__name__], name, getattr(mylib_py, name)) # Direct C access cdef get_a(): return mylib_c.CONST_A Where the Python access would be the new feature, and list all cpdef-ed stuff. However, from Parsing.py:2369: error(pos, "C struct/union/enum cannot be declared cpdef") From pyrex_differences.rst: If a function is declared :keyword:`cpdef` it can be called from and overridden by both extension and normal python subclasses. I believe the reason that cpdef-ed enums and similar are currently illegal is confusion between "can be called from Python" and "can be overridden from Python". I think these should be just like methods already are, in that you can "override" a method by subclassing it, but not by rebinding the name in the base class: >>> import pyximport; pyximport.install() >>> import rectangle as R >>> r = R.Rectangle(1, 2, 3, 4) >>> r.area = lambda(self): r.x1' Traceback (most recent call last): File "", line 1, in AttributeError: 'rectangle.Rectangle' object attribute 'area' is read-only Where rectangle.pyx is a minorly patched version of the last example from early_binding_for_speed.rst [1] and `area` is a cpdef-ed method. Why can't enums share this handling, with the enum taking the place of the method and the enum's module taking the place of the class? After all, enums have a Python-side tyoe (int or long). Unions don't really have a Python parallel, but structs do, so long as you can select which attributes should have (writable) Python interfaces. If we change the struct declaration syntax to be closer to the `cdef class` declaration syntax: cpdef struct Foo: cpdef public int intA cpdef readonly int intB cdef void *ptr We would both declare the important members of the C struct (as we can already do in Cython) and also have Cython automatically generate a Python class wrapping the struct (because of `cpdef struct`). The Python class would have: * Cython-generated getter/setter for intA (because of `cpdef public`) using the standard Python<->int coercion. * Similar Cython-generated getter for int B (because of `cpdef readonly`). * No Python access to ptr (standard C-access still possible through Cython). Doing something crazy like `cdef public void *ptr` would raise a compile-time error. I'm definately willing to help out with this (if someone will point me in the right direction), as the enum stuff would fix my original problem, and the struct stuff would allow me to rip out a bunch of boilerplate like cdef class Foo (object): cdef mylib.Foo _Foo def _intA_get(self): return self._Foo.intA def _intA_set(self, value): self._Foo.intA = value intA = property(fget=_intA_get, fset=_intA_set) def _intB_get(self): return self._Foo.intB intB = property(fget=_intB_get) from my wrapper code. Thanks, Trevor [1] While testing my overriding method example, I found a small typo in cython-docs' early_binding_for_speed.rst. Patch attached. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- From 80a52b3c0224e73a969b88fb414b6d026029a85e Mon Sep 17 00:00:00 2001 From: W. Trevor King Date: Thu, 17 Feb 2011 07:57:11 -0500 Subject: [PATCH] `int` -> `cdef int` when declaring local variables in early binding example. --- src/userguide/early_binding_for_speed.rst | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/userguide/early_binding_for_speed.rst b/src/userguide/early_binding_for_speed.rst index 07e0047..d44498d 100644 --- a/src/userguide/early_binding_for_speed.rst +++ b/src/userguide/early_binding_for_speed.rst @@ -53,7 +53,7 @@ where calls occur within Cython code. For example: def __init__(self, int x0, int y0, int x1, int y1): self.x0 = x0; self.y0 = y0; self.x1 = x1; self.y1 = y1 cdef int _area(self): - int area + cdef int area area = (self.x1 - self.x0) * (self.y1 - self.y0) if area < 0: area = -area @@ -88,7 +88,7 @@ overheads. Consider this code: def __init__(self, int x0, int y0, int x1, int y1): self.x0 = x0; self.y0 = y0; self.x1 = x1; self.y1 = y1 cpdef int area(self): - int area + cdef int area area = (self.x1 - self.x0) * (self.y1 - self.y0) if area < 0: area = -area -- 1.7.3.4 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From stefan_ml at behnel.de Thu Feb 17 15:09:15 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 17 Feb 2011 15:09:15 +0100 Subject: [Cython] Outdated `hg export` on cython-devel homepage In-Reply-To: <20110217135113.GB9353@tyr.home.net> References: <20110217135113.GB9353@tyr.home.net> Message-ID: <4D5D2C0B.6030508@behnel.de> W. Trevor King, 17.02.2011 14:51: > The `hg export` on the cython-devel page [1] should probably be > changed to (or additionally list) `git format-patch`, now that > Cython's versioned in Git. Keeping the `hg export` reference might be > useful for Mercurial lovers using hg-git. > > [1]: http://mail.python.org/mailman/listinfo/cython-devel Fixed, thanks! Stefan From wking at drexel.edu Thu Feb 17 14:44:24 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 17 Feb 2011 08:44:24 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110217132940.GA28686@tyr.home.net> References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> Message-ID: <20110217134424.GA9353@tyr.home.net> On Thu, Feb 17, 2011 at 08:29:41AM -0500, W. Trevor King wrote: > cpdef struct Foo: > cpdef public int intA > cpdef readonly int intB > cdef void *ptr Oops, for consistency with classes, the variables declarations should read `cdef public` and and `cdef readonly`. Perhaps `cdef struct` too, to match `cdef class`? I get a bit confused, because for some things (functions, methods) `cpdef` adds a Python interface. For others (attributes) it's `cdef public/readonly`. There are even some things (classes), where a plain `cdef` is enough to provide a Python interface. Perhaps I am just missing some subtle distinction between the effects of the various incantations? -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From wking at drexel.edu Thu Feb 17 15:23:28 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 17 Feb 2011 09:23:28 -0500 Subject: [Cython] [PATCH] Add .gitignores to cython and cython-docs Message-ID: <20110217142327.GA14083@tyr.home.net> Here's a pair of patches doing just that. I also ignore *.c in cython, because all .c files are currently auto-generated. Perhaps that will not always be the case? If it seems to risky, feel free to leave that part out. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- From a3916ac0ac058e43c9aa75e0a66312618be73edf Mon Sep 17 00:00:00 2001 From: W. Trevor King Date: Thu, 17 Feb 2011 09:16:21 -0500 Subject: [PATCH] Add *.c to .hgignore and create analogous .gitignore. --- .gitignore | 15 +++++++++++++++ .hgignore | 1 + 2 files changed, 16 insertions(+), 0 deletions(-) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2187b82 --- /dev/null +++ b/.gitignore @@ -0,0 +1,15 @@ +*.pyc +*.c +*.swp + +Cython/Compiler/Lexicon.pickle +BUILD/ +build/ +dist/ +.coverage +*~ +*.orig +*.rej +*.dep + +tags diff --git a/.hgignore b/.hgignore index 6fa7131..e005c72 100644 --- a/.hgignore +++ b/.hgignore @@ -1,6 +1,7 @@ syntax: glob *.pyc +*.c *.swp Cython/Compiler/Lexicon.pickle -- 1.7.3.4 -------------- next part -------------- From bf5bf5d3874cdcf34940b634c56dbeb35faa4137 Mon Sep 17 00:00:00 2001 From: W. Trevor King Date: Thu, 17 Feb 2011 09:20:30 -0500 Subject: [PATCH 2/2] Create .gitignore analogous to current .hgignore. --- .gitignore | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..d431956 --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +*.pyc +*~ +.*.swp + +build/ +_build/ -- 1.7.3.4 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From dalcinl at gmail.com Thu Feb 17 15:32:28 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 17 Feb 2011 11:32:28 -0300 Subject: [Cython] Fixing NumPy support for Python 3 (Stefan, please help!) Message-ID: Stefan, what do you think about the patch below? This hunk is part of a series of fixes required to get numpy-dev working under Python 3.2. The root of the issue is that __cythonbufferdefaults__ keys&values end-up being "bytes" (this coercion is triggered in Interpreter.py). diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py index 5b339da..b72deef 100755 --- a/Cython/Compiler/ExprNodes.py +++ b/Cython/Compiler/ExprNodes.py @@ -12,6 +12,7 @@ cython.declare(error=object, warning=object, warn_once=object, Builtin=object, Symtab=object, Utils=object, find_coercion_error debug_disposal_code=object, debug_temp_alloc=object, debug_coerc +import sys import operator from Errors import error, warning, warn_once, InternalError, CompileError @@ -1136,6 +1137,8 @@ class StringNode(PyConstNode): return self.result_code def compile_time_value(self, env): + if sys.version_info[0] >= 3 and self.unicode_value: + return self.unicode_value return self.value -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From wking at drexel.edu Thu Feb 17 15:35:22 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 17 Feb 2011 09:35:22 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217134424.GA9353@tyr.home.net> Message-ID: <20110217143521.GB14083@tyr.home.net> On Thu, Feb 17, 2011 at 10:53:15AM -0300, Lisandro Dalcin wrote: > Cython could certainly support "cpdef struct", it is just a matter to > define a proposal and find a contributor to implement it :-) Is there a CEP template (a la PEPs 9 and 12) that should be discussed on the mailing list, or do I develop it free-form on the wiki [2]? p.s. should I be picking one of cython-dev at codespeak or cython-devel at python? Is the shift not yet official? [2]: http://wiki.cython.org/enhancements/ -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From dalcinl at gmail.com Thu Feb 17 15:49:29 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 17 Feb 2011 11:49:29 -0300 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110217143521.GB14083@tyr.home.net> References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217134424.GA9353@tyr.home.net> <20110217143521.GB14083@tyr.home.net> Message-ID: On 17 February 2011 11:35, W. Trevor King wrote: > On Thu, Feb 17, 2011 at 10:53:15AM -0300, Lisandro Dalcin wrote: >> Cython could certainly support "cpdef struct", it is just a matter to >> define a proposal and find a contributor to implement it :-) > > Is there a CEP template (a la PEPs 9 and 12) that should be discussed > on the mailing list, or do I develop it free-form on the wiki [2]? > I would just develop it free-form on the wiki > p.s. should I be picking one of cython-dev at codespeak or > cython-devel at python? ?Is the shift not yet official? > Use cython-devel at python.org PS: Do we really need a full CEP for this? Do any of you object "cpdef struct" to automatically create a wrapper extension type for exposing to Python? I think all wee need to discuss is how to implement __cinit__(), and what to do with slots you want to use in C but cannot be mapped to a Python type (like pointers). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Thu Feb 17 16:16:26 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 17 Feb 2011 16:16:26 +0100 Subject: [Cython] Fixing NumPy support for Python 3 (Stefan, please help!) In-Reply-To: References: Message-ID: <4D5D3BCA.5010904@behnel.de> Lisandro Dalcin, 17.02.2011 15:32: > Stefan, what do you think about the patch below? This hunk is part of > a series of fixes required to get numpy-dev working under Python 3.2. > The root of the issue is that __cythonbufferdefaults__ keys&values > end-up being "bytes" (this coercion is triggered in Interpreter.py). > > > diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py > index 5b339da..b72deef 100755 > --- a/Cython/Compiler/ExprNodes.py > +++ b/Cython/Compiler/ExprNodes.py > @@ -12,6 +12,7 @@ cython.declare(error=object, warning=object, warn_once=object, > Builtin=object, Symtab=object, Utils=object, find_coercion_error > debug_disposal_code=object, debug_temp_alloc=object, debug_coerc > > +import sys > import operator > > from Errors import error, warning, warn_once, InternalError, CompileError > @@ -1136,6 +1137,8 @@ class StringNode(PyConstNode): > return self.result_code > > def compile_time_value(self, env): > + if sys.version_info[0]>= 3 and self.unicode_value: You must use "self.unicode_value is not None" here, it may be the empty string. > + return self.unicode_value > return self.value Ok, that's a tricky one. Just because the compilation is running in Py3 doesn't mean that the correct compile time value is a Unicode string - we don't know what it'll be used for. Doing the above will do the wrong thing e.g. in this case: DEF const_x = "abc" cdef str x = const_x The problem is: it is broken already, returning self.value is wrong because it drops available type information by returning plain bytes instead of str. And simply returning self.unicode_value in Py3 doesn't fix that. Stefan From dalcinl at gmail.com Thu Feb 17 19:11:46 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 17 Feb 2011 15:11:46 -0300 Subject: [Cython] python 2.7/3.x and numpy-dev (Dag, I need a quick comment) Message-ID: I'm working on a patch to get old, recent, and dev NumPy working in 2.7/3.x. So far, I had success, but I still have two failures like the one pasted below. Dag, could you elaborate a bit about the purpose of __Pyx_BufFmt_CheckString() ? It is just a validity check for pep 3118 format strings? Do you expect the failure below to be hard to fix? Just in case, the format string that triggers the failure is: >>> memoryview(np.zeros((1,), dtype=np.dtype('b,i', align=False))).format 'T{b:f0:=i:f1:}' ====================================================================== FAIL: numpy_test () Doctest: numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/python/3.2/lib/python3.2/doctest.py", line 2113, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for numpy_test File "/u/dalcinl/Devel/Cython/cython/BUILD/run/c/numpy_test.cpython-32dm.so", line 1, in numpy_test ---------------------------------------------------------------------- File "/u/dalcinl/Devel/Cython/cython/BUILD/run/c/numpy_test.cpython-32dm.so", line 155, in numpy_test Failed example: print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) Exception raised: Traceback (most recent call last): File "/usr/local/python/3.2/lib/python3.2/doctest.py", line 1248, in __run compileflags, 1), test.globs) File "", line 1, in print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) File "numpy_test.pyx", line 404, in numpy_test.test_packed_align (numpy_test.c:6367) ValueError: Buffer packing mode currently only allowed at beginning of format string (this is a defect) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Thu Feb 17 22:25:10 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 17 Feb 2011 13:25:10 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110217132940.GA28686@tyr.home.net> References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> Message-ID: On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: > This thread is coming over to cython-dev (and the new cython-devel) > from cython-users because it turns out it will probably require > chaning the Cython code. ?To get everyone who hasn't been following on > cython-users up to speed, here's a summary of what I'm trying to do: > > That's what I was trying to give with this: > > On Wed, Feb 09, 2011 at 12:23:25PM -0500, W. Trevor King wrote: >> I'm wrapping an external C library with Cython, so I have `mylib.pxd`: >> >> ? cdef extern from 'mylib.h' >> ? ? ? enum: CONST_A >> ? ? ? enum: CONST_B >> ? ? ? ... >> >> where I declare each constant macro from the library's header `mylib.h`: >> >> ? #define CONST_A 1 >> ? #define CONST_B 2 >> ? ... >> >> Now I want to expose those constants in Python, so I have `expose.pyx`: >> >> ? cimport mylib >> >> ? CONST_A = mylib.CONST_A >> ? CONST_B = mylib.CONST_B >> ? ... >> >> But the last part seems pretty silly. ?I'd like to do something like >> >> ? cimport mylib >> ? import sys >> >> ? for name in dir(mylib): >> ? ? ? setattr(sys.modules[__name__], name, getattr(mylib, name)) >> >> which compiles fine, but fails to import with... > > Looking into the Cython internals, everything defined in mylib.pxd is > stored as `Entry`s in a `ModuleScope`, and... > > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: >> > What I'm missing is a way to bind the ModuleScope namespace to a name >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >> > name)` will work in expose.pyx. >> >> You have also hit into the thorny issue that .pxd files are used for >> many things. They may be pure C library declarations with no Python >> module backing, they may be declarations of (externally implemented) >> Python modules (such as numpy.pxd), or they may be declarations for >> Cython-implemented modules. > >> > It seems like it would be easier to generate some kind of wrapper >> > class (PxdModule?) for mylib when it is cimported (at compile time), >> > and then further interactions would take care of themselves (at run >> > time). >> >> Would such an object be created anew for every module that cimports >> the declaration file? > > Hmm, That doesn't sound very nice, does it. ?However, .pxd files > declaring C libraries have no Python-space presence, so that was my > initial idea. > >> I have toyed with the idea of subclassing the module object itself for >> better support of C-level attributes from the Python (and Cython) >> namespaces. > > Sorry, I don't understand "better support of C-level attributes". ?Can > you give an example? The extern cpdef declarations are an example of this. >> Here's another idea, what if extern blocks could contain cpdef >> declarations, which would automatically generate a Python-level >> wrappers for the declared members (if possible, otherwise an error)? > > Ah, this sounds good! ?Of the three .pxd roles you list above, > external Python modules (e.g. numpy) and Cython-implemented modules > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. > What's missing is a way to give (where possible) declarations of > external C libraries a Python presence. ?cpdef fills this hole nicely, > since its whole purpose is to expose Python interfaces to > C-based elements. In the case of external Python modules, I'm not so sure we want to monkey-patch our stuff in (and where would we do it--on the first import of a cimporting module?) > A side effect of this cpdef change would be that now even bare .pxd > files (no matching .pyx) would have a Python presence, Where would it live? Would we just create this module (in essence, acting as if there was an empty .pyx file sitting there as well)? On this note, it may be worth pursuing the idea of a "cython helper" module where common code and objects could live. > so You could do > something like > > ? cimport mylib as mylib_c > ? import mylib as mylib_py > ? import sys > > ? # Access through Python > ? for name in dir(mylib_py): > ? ? ? setattr(sys.modules[__name__], name, getattr(mylib_py, name)) I think this smells worse than "import *" > ? # Direct C access > ? cdef get_a(): > ? ? ? return mylib_c.CONST_A > > Where the Python access would be the new feature, and list all > cpdef-ed stuff. > > However, from Parsing.py:2369: > > ? ?error(pos, "C struct/union/enum cannot be declared cpdef") > > From pyrex_differences.rst: > > ? ?If a function is declared :keyword:`cpdef` it can be called from > ? ?and overridden by both extension and normal python subclasses. > > I believe the reason that cpdef-ed enums and similar are currently > illegal is confusion between "can be called from Python" and "can be > overridden from Python". The reason that error statement is there is because it had no meaning, so an error was better than just ignoring it. > I think these should be just like methods > already are, in that you can "override" a method by subclassing it, > but not by rebinding the name in the base class: > > ? ?>>> import pyximport; pyximport.install() > ? ?>>> import rectangle as R > ? ?>>> r = R.Rectangle(1, 2, 3, 4) > ? ?>>> r.area = lambda(self): r.x1' > ? ?Traceback (most recent call last): > ? ? ?File "", line 1, in > ? ?AttributeError: 'rectangle.Rectangle' object attribute 'area' is read-only > > Where rectangle.pyx is a minorly patched version of the last example > from early_binding_for_speed.rst [1] and `area` is a cpdef-ed method. > > Why can't enums share this handling, with the enum taking the place of > the method and the enum's module taking the place of the class? ?After > all, enums have a Python-side tyoe (int or long). ?Unions don't really > have a Python parallel, They can be a cdef class wrapping the union type. > but structs do, so long as you can select > which attributes should have (writable) Python interfaces. ?If we > change the struct declaration syntax to be closer to the `cdef class` > declaration syntax: > > ? ?cpdef struct Foo: > ? ? ? ?cpdef public int intA > ? ? ? ?cpdef readonly int intB > ? ? ? ?cdef void *ptr > > We would both declare the important members of the C struct (as we can > already do in Cython) and also have Cython automatically generate a > Python class wrapping the struct (because of `cpdef struct`). ?The > Python class would have: > > * Cython-generated getter/setter for intA (because of `cpdef public`) > ?using the standard Python<->int coercion. > * Similar Cython-generated getter for int B (because of `cpdef > ?readonly`). > * No Python access to ptr (standard C-access still possible through > ?Cython). > > Doing something crazy like `cdef public void *ptr` would raise a > compile-time error. Yes, all of the above was exactly what I was proposing. > I'm definately willing to help out with this (if someone will point me > in the right direction), That would be great. > as the enum stuff would fix my original > problem, and the struct stuff would allow me to rip out a bunch of > boilerplate like > > ? ?cdef class Foo (object): > ? ? ? ?cdef mylib.Foo _Foo > > ? ? ? ?def _intA_get(self): > ? ? ? ? ? ?return self._Foo.intA > ? ? ? ?def _intA_set(self, value): > ? ? ? ? ? ?self._Foo.intA = value > ? ? ? ?intA = property(fget=_intA_get, fset=_intA_set) > > ? ? ? ?def _intB_get(self): > ? ? ? ? ? ?return self._Foo.intB > ? ? ? ?intB = property(fget=_intB_get) > > from my wrapper code. > > Thanks, > Trevor > > [1] While testing my overriding method example, I found a small typo > in cython-docs' early_binding_for_speed.rst. ?Patch attached. Thanks. Applied. - Robert From wking at drexel.edu Fri Feb 18 00:12:37 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 17 Feb 2011 18:12:37 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> Message-ID: <20110217231235.GA641@tyr.home.net> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: > On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: > > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: > >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: > >> > What I'm missing is a way to bind the ModuleScope namespace to a name > >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, > >> > name)` will work in expose.pyx. > >> > >> You have also hit into the thorny issue that .pxd files are used for > >> many things. They may be pure C library declarations with no Python > >> module backing, they may be declarations of (externally implemented) > >> Python modules (such as numpy.pxd), or they may be declarations for > >> Cython-implemented modules. > >> > >> Here's another idea, what if extern blocks could contain cpdef > >> declarations, which would automatically generate a Python-level > >> wrappers for the declared members (if possible, otherwise an error)? > > > > Ah, this sounds good! Of the three .pxd roles you list above, > > external Python modules (e.g. numpy) and Cython-implemented modules > > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. > > What's missing is a way to give (where possible) declarations of > > external C libraries a Python presence. cpdef fills this hole nicely, > > since its whole purpose is to expose Python interfaces to > > C-based elements. > > In the case of external Python modules, I'm not so sure we want to > monkey-patch our stuff in I don't think any of the changes we are suggesting would require changes to existing code, so .pxd-s with external implementations wouldn't be affected unless they brough the changes upon themselves. > (and where would we do it--on the first import of a cimporting > module?) Compilation is an issue. I think that .pxd files should be able to be cythoned directly, since then they Cython can build any wrappers they request. If the file has a matching .pyx file, cythoning either one should compile both together, since they'll produce a single Python .so module. > > A side effect of this cpdef change would be that now even bare .pxd > > files (no matching .pyx) would have a Python presence, > > Where would it live? Would we just create this module (in essence, > acting as if there was an empty .pyx file sitting there as well)? On > this note, it may be worth pursuing the idea of a "cython helper" > module where common code and objects could live. I'm not sure exactly what you mean by "cython helper", but this sounds like my 'bare .pyx can create a Python .so module idea above. > > so You could do > > something like > > > > cimport mylib as mylib_c > > import mylib as mylib_py > > import sys > > > > # Access through Python > > for name in dir(mylib_py): > > setattr(sys.modules[__name__], name, getattr(mylib_py, name)) > > I think this smells worse than "import *" Aha, thanks ;). I was stuck in my old .pxd-files-don't-create-modules-by-themselves mindset. Obviously, once they do, any Python code can access the contents directly and I can throw out all this indirection. > > However, from Parsing.py:2369: > > > > error(pos, "C struct/union/enum cannot be declared cpdef") > > > > From pyrex_differences.rst: > > > > If a function is declared :keyword:`cpdef` it can be called from > > and overridden by both extension and normal python subclasses. > > > > I believe the reason that cpdef-ed enums and similar are currently > > illegal is confusion between "can be called from Python" and "can be > > overridden from Python". > > The reason that error statement is there is because it had no meaning, > so an error was better than just ignoring it. Why does it have no meaning? I understand that it's not implemented yet, but a cpdef-ed enum or struct seems just as valid an idea as a cpdef-ed method. > > Unions don't really have a Python parallel, > > They can be a cdef class wrapping the union type. But I would think coercion would be difficult. Unions are usually (in my limited experience) for "don't worry about the type, just make sure it fits in X bytes". How would union->Python conversion work? > > cpdef struct Foo: > > cpdef public int intA > > cpdef readonly int intB > > cdef void *ptr > > > > We would both declare the important members of the C struct (as we can > > already do in Cython) and also have Cython automatically generate a > > Python class wrapping the struct (because of `cpdef struct`). The > > Python class would have: > > > > * Cython-generated getter/setter for intA (because of `cpdef public`) > > using the standard Python<->int coercion. > > * Similar Cython-generated getter for int B (because of `cpdef > > readonly`). > > * No Python access to ptr (standard C-access still possible through > > Cython). > > > > Doing something crazy like `cdef public void *ptr` would raise a > > compile-time error. > > Yes, all of the above was exactly what I was proposing. > > > I'm definately willing to help out with this (if someone will point me > > in the right direction), > > That would be great. Ok, I think we're pretty much agreed ;). I think that the next step is to start working on implementations of: * Stand alone .pxd -> Python module * Extending class cdef/cdpef/public/readonly handling to cover enums, stucts, and possibly unions. Problems with me getting started now: * I don't know where I should start mucking about in the source. * I don't know how to handle things like dummy enums (perhaps by requiring all cdef-ed enums to be named). * I'm going to go watch a movie ;). It's good to be moving forward! -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Fri Feb 18 00:54:48 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 17 Feb 2011 15:54:48 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> Message-ID: Forgot reply-all... didin't we have this discussion before about making that the default for this list as it is by-far the most common desired behavior? On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: > On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: >>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: >>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: >>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name >>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >>> >> > name)` will work in expose.pyx. >>> >> >>> >> You have also hit into the thorny issue that .pxd files are used for >>> >> many things. They may be pure C library declarations with no Python >>> >> module backing, they may be declarations of (externally implemented) >>> >> Python modules (such as numpy.pxd), or they may be declarations for >>> >> Cython-implemented modules. >>> >> >>> >> Here's another idea, what if extern blocks could contain cpdef >>> >> declarations, which would automatically generate a Python-level >>> >> wrappers for the declared members (if possible, otherwise an error)? >>> > >>> > Ah, this sounds good! ?Of the three .pxd roles you list above, >>> > external Python modules (e.g. numpy) and Cython-implemented modules >>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. >>> > What's missing is a way to give (where possible) declarations of >>> > external C libraries a Python presence. ?cpdef fills this hole nicely, >>> > since its whole purpose is to expose Python interfaces to >>> > C-based elements. >>> >>> In the case of external Python modules, I'm not so sure we want to >>> monkey-patch our stuff in >> >> I don't think any of the changes we are suggesting would require >> changes to existing code, so .pxd-s with external implementations >> wouldn't be affected unless they brough the changes upon themselves. > > Say, in numpy.pxd, I have > > cdef extern from "...": > ? ?cpdef struct obscure_internal_struct: > ? ? ? ?... > > Do we add an "obscure_internal_struct" onto the (global) numpy module? > What if it conflicts with a (runtime) name? This is the issue I'm > bringing up. > >>> (and where would we do it--on the first import of a cimporting >>> module?) >> >> Compilation is an issue. ?I think that .pxd files should be able to be >> cythoned directly, since then they Cython can build any wrappers they >> request. ?If the file has a matching .pyx file, cythoning either one >> should compile both together, since they'll produce a single Python >> .so module. > > In this case, I think it may make more sense to consider cimporting > stuff from .pyx files if no .pxd file is present. In any case, this is > a big change. I don't like the idea of always creating such a module > (it may be empty, or have name conflicts) nor the idea of > conditionally compiling it (does one have to look at the contents and > really understand Cython to see if a Python shadow will be created?) > >>> > A side effect of this cpdef change would be that now even bare .pxd >>> > files (no matching .pyx) would have a Python presence, >>> >>> Where would it live? Would we just create this module (in essence, >>> acting as if there was an empty .pyx file sitting there as well)? On >>> this note, it may be worth pursuing the idea of a "cython helper" >>> module where common code and objects could live. >> >> I'm not sure exactly what you mean by "cython helper", but this sounds >> like my 'bare .pyx can create a Python .so module idea above. > > I'm thinking of a place to put, e.g. the generator and bind-able > function classes, which are now re-implemented in every module that > uses them. I think there will be more cases like this in the future > rather than less. C-level code could be #included and linked from > "global" stores as well. However, that's somewhat tangential. > >>> > so You could do >>> > something like >>> > >>> > ? cimport mylib as mylib_c >>> > ? import mylib as mylib_py >>> > ? import sys >>> > >>> > ? # Access through Python >>> > ? for name in dir(mylib_py): >>> > ? ? ? setattr(sys.modules[__name__], name, getattr(mylib_py, name)) >>> >>> I think this smells worse than "import *" >> >> Aha, thanks ;). ?I was stuck in my old >> .pxd-files-don't-create-modules-by-themselves mindset. ?Obviously, >> once they do, any Python code can access the contents directly and I >> can throw out all this indirection. >> >>> > However, from Parsing.py:2369: >>> > >>> > ? ?error(pos, "C struct/union/enum cannot be declared cpdef") >>> > >>> > From pyrex_differences.rst: >>> > >>> > ? ?If a function is declared :keyword:`cpdef` it can be called from >>> > ? ?and overridden by both extension and normal python subclasses. >>> > >>> > I believe the reason that cpdef-ed enums and similar are currently >>> > illegal is confusion between "can be called from Python" and "can be >>> > overridden from Python". >>> >>> The reason that error statement is there is because it had no meaning, >>> so an error was better than just ignoring it. >> >> Why does it have no meaning? ?I understand that it's not implemented >> yet, but a cpdef-ed enum or struct seems just as valid an idea as a >> cpdef-ed method. > > We're only deciding what the meaning is now. I agree it's a valid > idea, its just that no one had even considered it yet. (I'll also > concede that it's a much more natural idea for someone who came to the > language once cpdef was already implemented.) > >>> > Unions don't really have a Python parallel, >>> >>> They can be a cdef class wrapping the union type. >> >> But I would think coercion would be difficult. ?Unions are usually (in >> my limited experience) for "don't worry about the type, just make sure >> it fits in X bytes". ?How would union->Python conversion work? > > There would be a wrapping type, e.g. > > cdef class MyUnion: > ? ?cdef union_type value > > with a bunch of setters/getters for the values, just like there are > for structs. (In fact the same code would handle structs and unions). > > This is getting into the wrapper-generator territory, but I'm starting > to think for simple things that might be worth it. > >>> > ? ?cpdef struct Foo: >>> > ? ? ? ?cpdef public int intA >>> > ? ? ? ?cpdef readonly int intB >>> > ? ? ? ?cdef void *ptr >>> > >>> > We would both declare the important members of the C struct (as we can >>> > already do in Cython) and also have Cython automatically generate a >>> > Python class wrapping the struct (because of `cpdef struct`). ?The >>> > Python class would have: >>> > >>> > * Cython-generated getter/setter for intA (because of `cpdef public`) >>> > ?using the standard Python<->int coercion. >>> > * Similar Cython-generated getter for int B (because of `cpdef >>> > ?readonly`). >>> > * No Python access to ptr (standard C-access still possible through >>> > ?Cython). >>> > >>> > Doing something crazy like `cdef public void *ptr` would raise a >>> > compile-time error. >>> >>> Yes, all of the above was exactly what I was proposing. >>> >>> > I'm definately willing to help out with this (if someone will point me >>> > in the right direction), >>> >>> That would be great. >> >> Ok, I think we're pretty much agreed ;). ?I think that the next step >> is to start working on implementations of: >> >> * Stand alone .pxd -> Python module > > I'm not sure we're agreed on this one. > >> * Extending class cdef/cdpef/public/readonly handling to cover enums, >> ?stucts, and possibly unions. > > This seems like the best first step. > >> Problems with me getting started now: >> >> * I don't know where I should start mucking about in the source. > > I would start by (1) creating/declaring cdef classes for cdef > structs/unions/enums (not textually, within, e.g. > CStructOrUnionDefNode.analyse_declarations(...) in Nodes.py) and then > making sure they get added to the module's namespace (look at > ModuleNode.py where the classes get added). The other files to look at > would be Parsing.py, PyrexTypes.py and Symtab.py. Could either try to > patch up parsing first, but I think it'd be more effective to make > this the default during development, as then all the tests would > exercise your code, and it's probably both easier to dive in there and > would give you a better idea of how best to handle the parsing. > > Also, the creating of cpdef functions is a bit hackish (for historical > reasons) and something we want to clean up, so I wouldn't go to great > lengths to try and emulate that. > >> * I don't know how to handle things like dummy enums (perhaps by >> ?requiring all cdef-ed enums to be named). > > All enums in C are named. > >> * I'm going to go watch a movie ;). > > :) > >> It's good to be moving forward! > > Yep. You might want to consider creating a github fork. > > - Robert > From sccolbert at gmail.com Fri Feb 18 03:23:54 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Thu, 17 Feb 2011 21:23:54 -0500 Subject: [Cython] Python C-api ref count semantics Message-ID: What is the rule of thumb when declaring functions from python's C-api when comes to ref counting? If I define a test case like so: cdef extern from "Python.h": object PyWeakref_NewRef(object, object) object PyWeakref_GET_OBJECT(object) class Foo(object): pass cdef class Test: cdef object obj cdef object wr def __init__(self): self.obj = Foo() self.wr = PyWeakref_NewRef(self.obj, None) def get_ref(self): return PyWeakref_GET_OBJECT(self.wr) I get these random python fatal errors: In [8]: %timeit -n 1000 t.get_ref() 1000 loops, best of 3: 224 ns per loop In [9]: %timeit -n 1000 t.get_ref() Fatal Python error: deallocating None Abort trap However, if I redefine the macro signature and getter function to this: from cpython cimport PyObject cdef extern from "Python.h": object PyWeakref_NewRef(object, object) PyObject* PyWeakref_GET_OBJECT(object) class Foo(object): pass cdef class Test: cdef object obj cdef object wr def __init__(self): self.obj = Foo() self.wr = PyWeakref_NewRef(self.obj, None) def clear_obj(self): self.obj = None def get_ref(self): return PyWeakref_GET_OBJECT(self.wr) Then it runs without issue. I can other gather is has to due the incref/decref going on in the generated C code. Should be doing something on my end to manually manage ref counts when using the C-Api? -------------- next part -------------- An HTML attachment was scrubbed... URL: From wking at drexel.edu Fri Feb 18 05:38:42 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 17 Feb 2011 23:38:42 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> Message-ID: <20110218043841.GA4195@tyr.home.net> On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: > On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: >>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: >>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: >>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name >>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >>> >> > name)` will work in expose.pyx. >>> >> >>> >> You have also hit into the thorny issue that .pxd files are used for >>> >> many things. They may be pure C library declarations with no Python >>> >> module backing, they may be declarations of (externally implemented) >>> >> Python modules (such as numpy.pxd), or they may be declarations for >>> >> Cython-implemented modules. >>> >> >>> >> Here's another idea, what if extern blocks could contain cpdef >>> >> declarations, which would automatically generate a Python-level >>> >> wrappers for the declared members (if possible, otherwise an error)? >>> > >>> > Ah, this sounds good! Of the three .pxd roles you list above, >>> > external Python modules (e.g. numpy) and Cython-implemented modules >>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. >>> > What's missing is a way to give (where possible) declarations of >>> > external C libraries a Python presence. cpdef fills this hole nicely, >>> > since its whole purpose is to expose Python interfaces to >>> > C-based elements. >>> >>> In the case of external Python modules, I'm not so sure we want to >>> monkey-patch our stuff in >> >> I don't think any of the changes we are suggesting would require >> changes to existing code, so .pxd-s with external implementations >> wouldn't be affected unless they brough the changes upon themselves. > > Say, in numpy.pxd, I have > > cdef extern from "...": > cpdef struct obscure_internal_struct: > ... > > Do we add an "obscure_internal_struct" onto the (global) numpy module? > What if it conflicts with a (runtime) name? This is the issue I'm > bringing up. Defining a cpdef *and* a non-matching external implementation should raise a compile-time error. I agree that there is a useful distinction between external-C-library and external-Python-module .pxd wrappers. Perhaps your matching blank .py or .pyx file could serve as a marker that the .pxd file should be inflated into its own full fledged python module. I'm not even sure how you would go about adding attributes to the numpy module. When/how would the Cython-created attributes get added? In the external-C-library case, if you define something incompatible in the matching .pyx or .py file, Cython will be able to see it and die with an appropriate error, so you can resolve your programming mistake. If you try to override anything in a .so compiled module at runtime, you'd get the same kind of error you currently do trying to rebind a compiled class' method. >>> (and where would we do it--on the first import of a cimporting >>> module?) >> >> Compilation is an issue. I think that .pxd files should be able to be >> cythoned directly, since then they Cython can build any wrappers they >> request. If the file has a matching .pyx file, cythoning either one >> should compile both together, since they'll produce a single Python >> .so module. > > In this case, I think it may make more sense to consider cimporting > stuff from .pyx files if no .pxd file is present. Can you cimport .pyx files that lack matching .pxd files? > In any case, this is a big change. I don't like the idea of always > creating such a module (it may be empty, or have name conflicts) It shouldn't take to long to compile an empty module ;). And odds are noone will waste time importing it either. > nor the idea of conditionally compiling it (does one have to look at > the contents and really understand Cython to see if a Python shadow > will be created?) I agree here. Under the mantra "explicit is better than implicit", we could have users add something like cdef module "modname" to any .pxd files that should be inflated into Python modules. .pxd files without such a tag would receive the current treatment, error on any cpdef, etc. The drawback of this approach is that it makes Cython more complicated, but if both behaviors are reasonable, there's probably no getting around that. >>> > A side effect of this cpdef change would be that now even bare .pxd >>> > files (no matching .pyx) would have a Python presence, >>> >>> Where would it live? Would we just create this module (in essence, >>> acting as if there was an empty .pyx file sitting there as well)? On >>> this note, it may be worth pursuing the idea of a "cython helper" >>> module where common code and objects could live. >> >> I'm not sure exactly what you mean by "cython helper", but this sounds >> like my 'bare .pyx can create a Python .so module idea above. > > I'm thinking of a place to put, e.g. the generator and bind-able > function classes, which are now re-implemented in every module that > uses them. I think there will be more cases like this in the future > rather than less. C-level code could be #included and linked from > "global" stores as well. However, that's somewhat tangential. That sounds more and more like per-pxd .so modules ;). >>> > Unions don't really have a Python parallel, >>> >>> They can be a cdef class wrapping the union type. >> >> But I would think coercion would be difficult. Unions are usually (in >> my limited experience) for "don't worry about the type, just make sure >> it fits in X bytes". How would union->Python conversion work? > > There would be a wrapping type, e.g. > > cdef class MyUnion: > cdef union_type value > > with a bunch of setters/getters for the values, just like there are > for structs. (In fact the same code would handle structs and unions). > > This is getting into the wrapper-generator territory, but I'm starting > to think for simple things that might be worth it. I think that if Cython will automatically generate a wrapper for cdef public int x it should generate a wrapper for cdef struct X: cdef public int x There really aren't that metatypes in C, so it doesn't seem like a slippery slope to me. Maybe I'm just missing something... >> Ok, I think we're pretty much agreed ;). I think that the next step >> is to start working on implementations of: >> >> * Stand alone .pxd -> Python module > > I'm not sure we're agreed on this one. Ok, it can wait while we hash out the best approach. >> * Extending class cdef/cdpef/public/readonly handling to cover enums, >> stucts, and possibly unions. > > This seems like the best first step. > >> Problems with me getting started now: >> >> * I don't know where I should start mucking about in the source. > > I would start by... Excellent. Thanks. >> * I don't know how to handle things like dummy enums (perhaps by >> requiring all cdef-ed enums to be named). > > All enums in C are named. But my Cython declaration (exposing a C `#define CONST_A 1`): cdef extern from 'mylib.h': enum: CONST_A is not a named enum. > Yep. You might want to consider creating a github fork. I've started a branch 'cdef-enums-stucts-and-unions' (no activity yet). Repo: http://www.physics.drexel.edu/~wking/code/git/cython.git Gitweb interface: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git I can put this up on GitHub if you want, but it's easier for me to self-host, and `git fetch` works just as well either way ;). -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From stefan_ml at behnel.de Fri Feb 18 05:50:31 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Feb 2011 05:50:31 +0100 Subject: [Cython] Gmane archive Message-ID: <4D5DFA97.4010709@behnel.de> Hi, I still didn't get a response from Gmane, but this article doesn't look promising: http://article.gmane.org/gmane.discuss/13987 So I guess we'll have to request a new group. Very unfortunate. Stefan From stefan_ml at behnel.de Fri Feb 18 05:54:48 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Feb 2011 05:54:48 +0100 Subject: [Cython] Python C-api ref count semantics In-Reply-To: References: Message-ID: <4D5DFB98.1020108@behnel.de> Chris Colbert, 18.02.2011 03:23: > What is the rule of thumb when declaring functions from python's C-api when > comes to ref counting? The general rule is to not declare them yourself. Instead, cimport them from the cpython package. (See Cython/Includes/) > If I define a test case like so: > > cdef extern from "Python.h": > > object PyWeakref_NewRef(object, object) > object PyWeakref_GET_OBJECT(object) > > > class Foo(object): > pass > > > cdef class Test: > > cdef object obj > cdef object wr > > def __init__(self): > self.obj = Foo() > self.wr = PyWeakref_NewRef(self.obj, None) > > def get_ref(self): > return PyWeakref_GET_OBJECT(self.wr) > > > I get these random python fatal errors: > > In [8]: %timeit -n 1000 t.get_ref() > 1000 loops, best of 3: 224 ns per loop > > In [9]: %timeit -n 1000 t.get_ref() > Fatal Python error: deallocating None > Abort trap > > > However, if I redefine the macro signature and getter function to this: > > from cpython cimport PyObject > > cdef extern from "Python.h": > > object PyWeakref_NewRef(object, object) > PyObject* PyWeakref_GET_OBJECT(object) > > > class Foo(object): > pass > > > cdef class Test: > > cdef object obj > cdef object wr > > def __init__(self): > self.obj = Foo() > self.wr = PyWeakref_NewRef(self.obj, None) > > def clear_obj(self): > self.obj = None > > def get_ref(self): > returnPyWeakref_GET_OBJECT(self.wr) > > > Then it runs without issue. I can other gather is has to due the > incref/decref going on in the generated C code. Should be doing something on > my end to manually manage ref counts when using the C-Api? Check the CPython documentation. Whenever a function returns a borrowed reference, you must declare it as PyObject* and cast it to . That being said, support for borrowed references has been long on the list but no-one has shown interest in doing it (or getting it done) so far. Stefan From stefan_ml at behnel.de Fri Feb 18 06:01:08 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Feb 2011 06:01:08 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> Message-ID: <4D5DFD14.2010808@behnel.de> Robert Bradshaw, 18.02.2011 00:54: > Forgot reply-all... didin't we have this discussion before about > making that the default for this list as it is by-far the most common > desired behavior? Yes we did. And I guess it would be "the default" for mailing lists if it was just that: a default, not something that breaks replying. However, given that this has been discussed and decided, I'll just go and break replying for this list again. Stefan From stefan_ml at behnel.de Fri Feb 18 06:19:32 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Feb 2011 06:19:32 +0100 Subject: [Cython] Fixing NumPy support for Python 3 (Stefan, please help!) In-Reply-To: References: <4D5D3BCA.5010904@behnel.de> Message-ID: <4D5E0164.8040107@behnel.de> Lisandro Dalcin, 17.02.2011 17:24: > On 17 February 2011 12:16, Stefan Behnel wrote: >> Lisandro Dalcin, 17.02.2011 15:32: >>> >>> Stefan, what do you think about the patch below? This hunk is part of >>> a series of fixes required to get numpy-dev working under Python 3.2. >>> The root of the issue is that __cythonbufferdefaults__ keys&values >>> end-up being "bytes" (this coercion is triggered in Interpreter.py). >>> >>> >>> diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py >>> index 5b339da..b72deef 100755 >>> --- a/Cython/Compiler/ExprNodes.py >>> +++ b/Cython/Compiler/ExprNodes.py >>> @@ -12,6 +12,7 @@ cython.declare(error=object, warning=object, >>> warn_once=object, >>> Builtin=object, Symtab=object, Utils=object, >>> find_coercion_error >>> debug_disposal_code=object, debug_temp_alloc=object, >>> debug_coerc >>> >>> +import sys >>> import operator >>> >>> from Errors import error, warning, warn_once, InternalError, CompileError >>> @@ -1136,6 +1137,8 @@ class StringNode(PyConstNode): >>> return self.result_code >>> >>> def compile_time_value(self, env): >>> + if sys.version_info[0]>= 3 and self.unicode_value: >> >> You must use "self.unicode_value is not None" here, it may be the empty >> string. >> >>> + return self.unicode_value >>> return self.value >> >> Ok, that's a tricky one. Just because the compilation is running in Py3 >> doesn't mean that the correct compile time value is a Unicode string - we >> don't know what it'll be used for. > > OK, I've found an alternative workaround. What do you think? > > diff --git a/Cython/Compiler/Interpreter.py b/Cython/Compiler/Interpreter.py > index 83cb184..9fb5fe5 100644 > --- a/Cython/Compiler/Interpreter.py > +++ b/Cython/Compiler/Interpreter.py > @@ -6,6 +6,7 @@ For now this only covers parse tree to value conversion of > compile-time values. > """ > > +import sys > from Nodes import * > from ExprNodes import * > from Errors import CompileError > @@ -44,6 +45,10 @@ def interpret_compiletime_options(optlist, optdict, > type_env=None, type_args=()) > else: > raise CompileError(node.pos, "Type not allowed here.") > else: > + if (sys.version_info[0]>=3 and > + isinstance(node, StringNode) and > + node.unicode_value is not None): > + return (node.unicode_value, node.pos) > return (node.compile_time_value(empty_scope), node.pos) > > if optlist: > @@ -52,6 +57,7 @@ def interpret_compiletime_options(optlist, optdict, > type_env=None, type_args=()) > assert isinstance(optdict, DictNode) > new_optdict = {} > for item in optdict.key_value_pairs: > - new_optdict[item.key.value] = interpret(item.value, item.key.value) > + new_key, dummy = interpret(item.key, None) > + new_optdict[new_key] = interpret(item.value, item.key.value) > optdict = new_optdict > return (optlist, new_optdict) This still isn't something that looks right. It just does the same thing at a different place. Actually, I'm not sure there is a way to "get it right". >> Doing the above will do the wrong thing >> e.g. in this case: >> >> DEF const_x = "abc" >> cdef str x = const_x >> >> The problem is: it is broken already, returning self.value is wrong because >> it drops available type information by returning plain bytes instead of str. >> And simply returning self.unicode_value in Py3 doesn't fix that. > > I see... So the correct compile time value for StringNode should > depend on options.language_level, right? Hmmm. I'm not sure that would solve it. Py2 str has the property of changing type depending on the runtime environment. So this is actually independent of the language_level (-3 has easy semantics here). I mean, it won't even solve the problem at hand, because the code could still be Py2 but require a unicode string value because it gets *compiled* under Python 3. It shouldn't depend on the compile time environment at all, but the NumPy problem shows that it has to sometimes. I think we have to find a way to keep the double bytes/unicode string identity alive during runtime processing, up to the point where we can (or have to) decide what to make of it. Stefan From animator333 at yahoo.com Fri Feb 18 06:46:53 2011 From: animator333 at yahoo.com (Prashant Saxena) Date: Fri, 18 Feb 2011 11:16:53 +0530 (IST) Subject: [Cython] del command Message-ID: <741488.46946.qm@web94905.mail.in2.yahoo.com> Hello, python 2.6.6 Cython-0.14.1 xp, mac, linux When I am using "del" command in my scripts and compiling using pure python mode, I am getting an error: "Deletion of local or C global name not supported" There are couple of places where I have to delete an object. I am using a hack to solve this but I am not sure it's correct way. In a simple python script (delobj.py), I am writing a function: def deleteObject(obj): del obj I am importing this script in all the scripts I am compiling using cython and there I am using: delobj.deleteObject(obj) this is working fine and no errors. Is this is the correct way? Regards Prashant From stefan_ml at behnel.de Fri Feb 18 07:07:53 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Feb 2011 07:07:53 +0100 Subject: [Cython] del command In-Reply-To: <741488.46946.qm@web94905.mail.in2.yahoo.com> References: <741488.46946.qm@web94905.mail.in2.yahoo.com> Message-ID: <4D5E0CB9.7050603@behnel.de> Reply sent to cython-users. Stefan From dagss at student.matnat.uio.no Fri Feb 18 09:13:36 2011 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 18 Feb 2011 09:13:36 +0100 Subject: [Cython] python 2.7/3.x and numpy-dev (Dag, I need a quick comment) In-Reply-To: References: Message-ID: <4D5E2A30.9040601@student.matnat.uio.no> On 02/17/2011 07:11 PM, Lisandro Dalcin wrote: > I'm working on a patch to get old, recent, and dev NumPy working in > 2.7/3.x. So far, I had success, but I still have two failures like the > one pasted below. > > Dag, could you elaborate a bit about the purpose of > __Pyx_BufFmt_CheckString() ? It is just a validity check for pep 3118 > format strings? Do you expect the failure below to be hard to fix?s. Yes, it compares the string with the RTTI that is saved for the type that is expected by the code. I remember there was something about the '=' specifier that meant it was not completely trivial to fix, but still, it's just about doing it. The code in question is a bit convoluted; the reason I did that is because I wanted to allow things to match if the binary layouts matched even if the 'struct structure' wasn't the same... As for user code, a quick hack around this is 'cast=True' in the buffer spec. Please file a bug... Dag Sverre > Just in case, the format string that triggers the failure is: > >>>> memoryview(np.zeros((1,), dtype=np.dtype('b,i', align=False))).format > 'T{b:f0:=i:f1:}' > > > ====================================================================== > FAIL: numpy_test () > Doctest: numpy_test > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/usr/local/python/3.2/lib/python3.2/doctest.py", line 2113, in runTest > raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for numpy_test > File "/u/dalcinl/Devel/Cython/cython/BUILD/run/c/numpy_test.cpython-32dm.so", > line 1, in numpy_test > > ---------------------------------------------------------------------- > File "/u/dalcinl/Devel/Cython/cython/BUILD/run/c/numpy_test.cpython-32dm.so", > line 155, in numpy_test > Failed example: > print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) > Exception raised: > Traceback (most recent call last): > File "/usr/local/python/3.2/lib/python3.2/doctest.py", line 1248, in __run > compileflags, 1), test.globs) > File "", line 1, in > print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', > align=False)))) > File "numpy_test.pyx", line 404, in numpy_test.test_packed_align > (numpy_test.c:6367) > ValueError: Buffer packing mode currently only allowed at > beginning of format string (this is a defect) > > From lilotom at gmail.com Fri Feb 18 10:10:09 2011 From: lilotom at gmail.com (Thomson) Date: Fri, 18 Feb 2011 17:10:09 +0800 Subject: [Cython] How to update python grammar in Windows? Message-ID: I tried to add a statement to Python, but I don't know how to update the grammar header files such as graminit.c&graminit.h after I updated the grammar file (Garmmar/Garmmar). It seems pgenmain.c could do this, but these is no pgen.exe in the build output of Visual Studio. How can I update the grammar files in Windows? Thanks, Thomson From stefan_ml at behnel.de Fri Feb 18 10:17:43 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Feb 2011 10:17:43 +0100 Subject: [Cython] How to update python grammar in Windows? In-Reply-To: References: Message-ID: <4D5E3937.60105@behnel.de> Thomson, 18.02.2011 10:10: > I tried to add a statement to Python, but I don't know how to update > the grammar header files such as graminit.c&graminit.h after I updated > the grammar file (Garmmar/Garmmar). It seems pgenmain.c could do this, > but these is no pgen.exe in the build output of Visual Studio. How can > I update the grammar files in Windows? Wrong list, actually doubly so. This is the core developers mailing list of the Cython project. Welcome! :) You may want to ask your question on comp.lang.python (or "python-list at python.org"). Or maybe the python-dev list on python.org, in case it's really CPython core development related (in which case you should be subscribed to python-dev anyway). But I'd try c.l.py first. Stefan From stefan_ml at behnel.de Fri Feb 18 16:23:32 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Feb 2011 16:23:32 +0100 Subject: [Cython] Fixing NumPy support for Python 3 (Stefan, please help!) In-Reply-To: References: <4D5D3BCA.5010904@behnel.de> <4D5E0164.8040107@behnel.de> Message-ID: <4D5E8EF4.4070800@behnel.de> Lisandro Dalcin, 18.02.2011 15:38: > On 18 February 2011 02:19, Stefan Behnel wrote: >> Lisandro Dalcin, 17.02.2011 17:24: >>> >>> >>> OK, I've found an alternative workaround. What do you think? >>> >>> diff --git a/Cython/Compiler/Interpreter.py >>> b/Cython/Compiler/Interpreter.py >>> index 83cb184..9fb5fe5 100644 >>> --- a/Cython/Compiler/Interpreter.py >>> +++ b/Cython/Compiler/Interpreter.py >>> @@ -6,6 +6,7 @@ For now this only covers parse tree to value conversion of >>> compile-time values. >>> """ >>> >>> +import sys >>> from Nodes import * >>> from ExprNodes import * >>> from Errors import CompileError >>> @@ -44,6 +45,10 @@ def interpret_compiletime_options(optlist, optdict, >>> type_env=None, type_args=()) >>> else: >>> raise CompileError(node.pos, "Type not allowed here.") >>> else: >>> + if (sys.version_info[0]>=3 and >>> + isinstance(node, StringNode) and >>> + node.unicode_value is not None): >>> + return (node.unicode_value, node.pos) >>> return (node.compile_time_value(empty_scope), node.pos) >>> >>> if optlist: >>> @@ -52,6 +57,7 @@ def interpret_compiletime_options(optlist, optdict, >>> type_env=None, type_args=()) >>> assert isinstance(optdict, DictNode) >>> new_optdict = {} >>> for item in optdict.key_value_pairs: >>> - new_optdict[item.key.value] = interpret(item.value, >>> item.key.value) >>> + new_key, dummy = interpret(item.key, None) >>> + new_optdict[new_key] = interpret(item.value, item.key.value) >>> optdict = new_optdict >>> return (optlist, new_optdict) >> >> This still isn't something that looks right. It just does the same thing at >> a different place. Actually, I'm not sure there is a way to "get it right". >> > > IIUC, the purpose of that code is to bring stuff from .pyx code to > Python world at Cython runtime, that code is trying to "coerce" a > DictNode object to a regular Python dict to be used later for > implementing compiler directives. So, if your pyx/pxd says > {"mode":"strided"} , and that is going to be compiler directive > options, you need to map StringNode -> __builtin__.str (and then use > node.value in Py3 or node.unicode_value in Py3). > > Stefan, look at numpy.pxd, you will see $cdef __cythonbufferdefaults__ > = {"mode": "strided"}$. W need to "convert" that declaration in a > regular Python dict to be used in Buffer.py, in such a way that > $"mode" in option_dict$ or $option_dict["mode"]$ work. Currently, in > Python 3, the key,value pair endup being bytes, then things fails... > > I still think that what I'm doing is not so bad. After all, I'm > hacking a method called interpret_compiletime_options(), the purpose > of its output is not code generation. Hmm, right. I guess that'll do then. Please go ahead and push this in. If it helps NumPy users in Py3, it's worth getting out rather sooner than later. Stefan From lists at onerussian.com Fri Feb 18 19:31:34 2011 From: lists at onerussian.com (Yaroslav Halchenko) Date: Fri, 18 Feb 2011 13:31:34 -0500 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: <20110217123840.GF21658@onerussian.com> References: <4D5B6A94.6050707@behnel.de> <20110217123840.GF21658@onerussian.com> Message-ID: <20110218183134.GP21658@onerussian.com> And here are the first fruits: unittests fail on some platforms (also I have forgotten that $HOME is not guarantee to be writable, but that is a pure packaging-related side-point): All logs are available from https://buildd.debian.org/build.cgi?pkg=cython&dist=sid armel, powerpc, s390 (hppa is not yet done but probably the same) -- big endians: e.g. https://buildd.debian.org/fetch.cgi?pkg=cython;ver=0.14.1-3;arch=armel;stamp=1298052709 I could be wrong but it seems they all boil down to FAIL: Doctest: dictintindex.__test__.test_get_char_neg (line 1) so the others fail as well, e.g. FAIL: Doctest: c_int_types_T255 FAIL: Doctest: c_int_types_T255.__test__.test_char (line 90) FAIL: Doctest: dictintindex.__test__.test_get_char_neg (line 1) otherwise -- amazingly nice ;-) On Thu, 17 Feb 2011, Yaroslav Halchenko wrote: > > > | AssertionError > > > `--- > > > what could be done about it or should it be excluded? > > I've pushed some fixes. Now this testcase should run from ancient > > Python 2.3 to head Python 3.2, both for static and sharedlib builds > > (but not in Windows). > first of all THANKS for the patch -- I picked it up into 0.14.1-2 debian > package. Now tests are enabled and I just uploaded 0.14.1-2 into Debian > -- lets see how it would go across architectures ;-) > 2nd -- THANKS for ...: at first I got confused why I saw no commits > since Dec in HG clone of Cython I had, and then mentioned that you > moved to using GIT and github. Awesome and thank you for taking care > about my sanity (although unintentionally I guess ;) ) -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic From dalcinl at gmail.com Fri Feb 18 22:33:24 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 18 Feb 2011 18:33:24 -0300 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: <20110218205816.GT21658@onerussian.com> References: <4D5B6A94.6050707@behnel.de> <20110217123840.GF21658@onerussian.com> <20110218183134.GP21658@onerussian.com> <20110218205816.GT21658@onerussian.com> Message-ID: 2011/2/18 Yaroslav Halchenko : > > awesome! ?I will test it asap -- just let me know if there would be > another commit for above or I should tune up $HOME ;) > https://github.com/cython/cython/commit/475e9a085654252d5a274dab2118b746e8bda4eb >> We are very near to be crystal clear... > > yeap > And now we should be crystal clear, finally! Could you confirm? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Fri Feb 18 23:08:04 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 18 Feb 2011 14:08:04 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110218043841.GA4195@tyr.home.net> References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> <20110218043841.GA4195@tyr.home.net> Message-ID: On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote: > On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: >> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >>> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: >>>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: >>>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: >>>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: >>>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name >>>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, >>>> >> > name)` will work in expose.pyx. >>>> >> >>>> >> You have also hit into the thorny issue that .pxd files are used for >>>> >> many things. They may be pure C library declarations with no Python >>>> >> module backing, they may be declarations of (externally implemented) >>>> >> Python modules (such as numpy.pxd), or they may be declarations for >>>> >> Cython-implemented modules. >>>> >> >>>> >> Here's another idea, what if extern blocks could contain cpdef >>>> >> declarations, which would automatically generate a Python-level >>>> >> wrappers for the declared members (if possible, otherwise an error)? >>>> > >>>> > Ah, this sounds good! ?Of the three .pxd roles you list above, >>>> > external Python modules (e.g. numpy) and Cython-implemented modules >>>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. >>>> > What's missing is a way to give (where possible) declarations of >>>> > external C libraries a Python presence. ?cpdef fills this hole nicely, >>>> > since its whole purpose is to expose Python interfaces to >>>> > C-based elements. >>>> >>>> In the case of external Python modules, I'm not so sure we want to >>>> monkey-patch our stuff in >>> >>> I don't think any of the changes we are suggesting would require >>> changes to existing code, so .pxd-s with external implementations >>> wouldn't be affected unless they brough the changes upon themselves. >> >> Say, in numpy.pxd, I have >> >> cdef extern from "...": >> ? ?cpdef struct obscure_internal_struct: >> ? ? ? ?... >> >> Do we add an "obscure_internal_struct" onto the (global) numpy module? >> What if it conflicts with a (runtime) name? This is the issue I'm >> bringing up. > > Defining a cpdef *and* a non-matching external implementation should > raise a compile-time error. ?I agree that there is a useful > distinction between external-C-library and external-Python-module .pxd > wrappers. ?Perhaps your matching blank .py or .pyx file could serve as > a marker that the .pxd file should be inflated into its own full > fledged python module. ?I'm not even sure how you would go about > adding attributes to the numpy module. ?When/how would the > Cython-created attributes get added? Yes, this is exactly the issue. > In the external-C-library case, if you define something incompatible > in the matching .pyx or .py file, Cython will be able to see it and > die with an appropriate error, so you can resolve your programming > mistake. That is only if you have a matching .pyx of .py file. Of course, there could be a module by that name in the user's runtime environment that we don't know about, but that currently is not in conflict with a .pxd only file. > If you try to override anything in a .so compiled module at runtime, > you'd get the same kind of error you currently do trying to rebind a > compiled class' method. That's the desired behavior for statically-bound globals, but implementing it is not so trivial. >>>> (and where would we do it--on the first import of a cimporting >>>> module?) >>> >>> Compilation is an issue. ?I think that .pxd files should be able to be >>> cythoned directly, since then they Cython can build any wrappers they >>> request. ?If the file has a matching .pyx file, cythoning either one >>> should compile both together, since they'll produce a single Python >>> .so module. >> >> In this case, I think it may make more sense to consider cimporting >> stuff from .pyx files if no .pxd file is present. > > Can you cimport .pyx files that lack matching .pxd files? > >> In any case, this is a big change. I don't like the idea of always >> creating such a module (it may be empty, or have name conflicts) > > It shouldn't take to long to compile an empty module ;). ?And odds are > noone will waste time importing it either. > >> nor the idea of conditionally compiling it (does one have to look at >> the contents and really understand Cython to see if a Python shadow >> will be created?) > > I agree here. > > Under the mantra "explicit is better than implicit", we could have > users add something like > > ? ?cdef module "modname" > > to any .pxd files that should be inflated into Python modules. ?.pxd > files without such a tag would receive the current treatment, error on > any cpdef, etc. ?The drawback of this approach is that it makes Cython > more complicated, but if both behaviors are reasonable, there's > probably no getting around that. The other drawback is that it subverts the usual filename <-> module name convention that one usually expects. >>>> > A side effect of this cpdef change would be that now even bare .pxd >>>> > files (no matching .pyx) would have a Python presence, >>>> >>>> Where would it live? Would we just create this module (in essence, >>>> acting as if there was an empty .pyx file sitting there as well)? On >>>> this note, it may be worth pursuing the idea of a "cython helper" >>>> module where common code and objects could live. >>> >>> I'm not sure exactly what you mean by "cython helper", but this sounds >>> like my 'bare .pyx can create a Python .so module idea above. >> >> I'm thinking of a place to put, e.g. the generator and bind-able >> function classes, which are now re-implemented in every module that >> uses them. I think there will be more cases like this in the future >> rather than less. C-level code could be #included and linked from >> "global" stores as well. However, that's somewhat tangential. > > That sounds more and more like per-pxd .so modules ;). > >>>> > Unions don't really have a Python parallel, >>>> >>>> They can be a cdef class wrapping the union type. >>> >>> But I would think coercion would be difficult. ?Unions are usually (in >>> my limited experience) for "don't worry about the type, just make sure >>> it fits in X bytes". ?How would union->Python conversion work? >> >> There would be a wrapping type, e.g. >> >> cdef class MyUnion: >> ? ?cdef union_type value >> >> with a bunch of setters/getters for the values, just like there are >> for structs. (In fact the same code would handle structs and unions). >> >> This is getting into the wrapper-generator territory, but I'm starting >> to think for simple things that might be worth it. > > I think that if Cython will automatically generate a wrapper for > > ? ?cdef public int x > > it should generate a wrapper for > > ? ?cdef struct X: cdef public int x Or cdef public struct X: int x readonly int z private int z I would perhaps say that non-Pythonable non-private members in public structs would be a compile error. > There really aren't that metatypes in C, so it doesn't seem like a > slippery slope to me. ?Maybe I'm just missing something... > >>> Ok, I think we're pretty much agreed ;). ?I think that the next step >>> is to start working on implementations of: >>> >>> * Stand alone .pxd -> Python module >> >> I'm not sure we're agreed on this one. > > Ok, it can wait while we hash out the best approach. > >>> * Extending class cdef/cdpef/public/readonly handling to cover enums, >>> ?stucts, and possibly unions. >> >> This seems like the best first step. >> >>> Problems with me getting started now: >>> >>> * I don't know where I should start mucking about in the source. >> >> I would start by... > > Excellent. ?Thanks. > >>> * I don't know how to handle things like dummy enums (perhaps by >>> ?requiring all cdef-ed enums to be named). >> >> All enums in C are named. > > But my Cython declaration (exposing a C `#define CONST_A 1`): > > ? ?cdef extern from 'mylib.h': > ? ? ? ?enum: CONST_A > > is not a named enum. Ah, yes. Maybe we require a name (that would only be used in Python space). >> Yep. You might want to consider creating a github fork. > > I've started a branch 'cdef-enums-stucts-and-unions' (no activity > yet). > > Repo: > ?http://www.physics.drexel.edu/~wking/code/git/cython.git > > Gitweb interface: > ?http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git > > I can put this up on GitHub if you want, but it's easier for me to > self-host, and `git fetch` works just as well either way ;). Code review is much easier if you fork off the public branch and do pull-requests, but being a DVCS you of course don't have to "choose" where your master repo lives. - Robert From lists at onerussian.com Fri Feb 18 23:15:18 2011 From: lists at onerussian.com (Yaroslav Halchenko) Date: Fri, 18 Feb 2011 17:15:18 -0500 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: References: <4D5B6A94.6050707@behnel.de> <20110217123840.GF21658@onerussian.com> <20110218183134.GP21658@onerussian.com> <20110218205816.GT21658@onerussian.com> Message-ID: <20110218221518.GU21658@onerussian.com> On Fri, 18 Feb 2011, Lisandro Dalcin wrote: > > awesome! ?I will test it asap -- just let me know if there would be > > another commit for above or I should tune up $HOME ;) > https://github.com/cython/cython/commit/475e9a085654252d5a274dab2118b746e8bda4eb evil you -- and who would address: Caller is responsible for deleting the directory when done with it. ;-) > >> We are very near to be crystal clear... > > yeap > And now we should be crystal clear, finally! Could you confirm? I will... manually first on one of those platforms... I will let you know of cause ;) -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic From dalcinl at gmail.com Fri Feb 18 23:42:59 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 18 Feb 2011 19:42:59 -0300 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: <20110218221518.GU21658@onerussian.com> References: <4D5B6A94.6050707@behnel.de> <20110217123840.GF21658@onerussian.com> <20110218183134.GP21658@onerussian.com> <20110218205816.GT21658@onerussian.com> <20110218221518.GU21658@onerussian.com> Message-ID: 2011/2/18 Yaroslav Halchenko : > > On Fri, 18 Feb 2011, Lisandro Dalcin wrote: >> > awesome! ?I will test it asap -- just let me know if there would be >> > another commit for above or I should tune up $HOME ;) >> https://github.com/cython/cython/commit/475e9a085654252d5a274dab2118b746e8bda4eb > > evil you -- and who would address: > ? ?Caller is responsible for deleting the directory when done with it. > ;-) > Well, I decided to not delete it just in case things go wrong and you need to check the output... I can easily fix that in a tearDown() method... Should I? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From lists at onerussian.com Sat Feb 19 01:11:36 2011 From: lists at onerussian.com (Yaroslav Halchenko) Date: Fri, 18 Feb 2011 19:11:36 -0500 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: References: <4D5B6A94.6050707@behnel.de> <20110217123840.GF21658@onerussian.com> <20110218183134.GP21658@onerussian.com> <20110218205816.GT21658@onerussian.com> <20110218221518.GU21658@onerussian.com> Message-ID: <20110219001136.GV21658@onerussian.com> > >> > awesome! ?I will test it asap -- just let me know if there would be > >> > another commit for above or I should tune up $HOME ;) > >> https://github.com/cython/cython/commit/475e9a085654252d5a274dab2118b746e8bda4eb > > evil you -- and who would address: > > ? ?Caller is responsible for deleting the directory when done with it. > > ;-) > Well, I decided to not delete it just in case things go wrong and you > need to check the output... I can easily fix that in a tearDown() > method... Should I? Well, that is up to you guys -- I do not think there is anything in Debian policy/guidelines about cleaning up TMPDIR. I, personally, would have preferred them being cleaned up. -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic From lists at onerussian.com Sat Feb 19 08:28:22 2011 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sat, 19 Feb 2011 02:28:22 -0500 Subject: [Cython] Fwd: Re: Cython builds on various Debian platforms In-Reply-To: <20110218221518.GU21658@onerussian.com> References: <4D5B6A94.6050707@behnel.de> <20110217123840.GF21658@onerussian.com> <20110218183134.GP21658@onerussian.com> <20110218205816.GT21658@onerussian.com> <20110218221518.GU21658@onerussian.com> Message-ID: <20110219072822.GZ21658@onerussian.com> confirming now -- built/tested fine manually on s390, uploaded, and now built across nearly all (few still building/testing) archs: https://buildd.debian.org/pkg.cgi?pkg=cython On Fri, 18 Feb 2011, Yaroslav Halchenko wrote: > > >> We are very near to be crystal clear... > > > yeap > > And now we should be crystal clear, finally! Could you confirm? > I will... manually first on one of those platforms... I will let you > know of cause ;) -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic From stefan_ml at behnel.de Sat Feb 19 10:24:05 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 19 Feb 2011 10:24:05 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> <20110218043841.GA4195@tyr.home.net> Message-ID: <4D5F8C35.4010309@behnel.de> Robert Bradshaw, 18.02.2011 23:08: > On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote: >> On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: >>> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >>>> Compilation is an issue. I think that .pxd files should be able to be >>>> cythoned directly, since then they Cython can build any wrappers they >>>> request. If the file has a matching .pyx file, cythoning either one >>>> should compile both together, since they'll produce a single Python >>>> .so module. >>> >>> In this case, I think it may make more sense to consider cimporting >>> stuff from .pyx files if no .pxd file is present. >> >> Can you cimport .pyx files that lack matching .pxd files? >> >>> In any case, this is a big change. I don't like the idea of always >>> creating such a module (it may be empty, or have name conflicts) >> >> It shouldn't take to long to compile an empty module ;). And odds are >> noone will waste time importing it either. >> >>> nor the idea of conditionally compiling it (does one have to look at >>> the contents and really understand Cython to see if a Python shadow >>> will be created?) >> >> I agree here. >> >> Under the mantra "explicit is better than implicit", we could have >> users add something like >> >> cdef module "modname" >> >> to any .pxd files that should be inflated into Python modules. .pxd >> files without such a tag would receive the current treatment, error on >> any cpdef, etc. The drawback of this approach is that it makes Cython >> more complicated, but if both behaviors are reasonable, there's >> probably no getting around that. > > The other drawback is that it subverts the usual filename<-> module > name convention that one usually expects. +1 >>>>>> A side effect of this cpdef change would be that now even bare .pxd >>>>>> files (no matching .pyx) would have a Python presence, >>>>> >>>>> Where would it live? Would we just create this module (in essence, >>>>> acting as if there was an empty .pyx file sitting there as well)? On >>>>> this note, it may be worth pursuing the idea of a "cython helper" >>>>> module where common code and objects could live. >>>> >>>> I'm not sure exactly what you mean by "cython helper", but this sounds >>>> like my 'bare .pyx can create a Python .so module idea above. >>> >>> I'm thinking of a place to put, e.g. the generator and bind-able >>> function classes, which are now re-implemented in every module that >>> uses them. I think there will be more cases like this in the future >>> rather than less. C-level code could be #included and linked from >>> "global" stores as well. However, that's somewhat tangential. If you generate more than one file from a .pyx, including files that are shared between compiler runs (or even readily built as .so files), you'd quickly end up in dependency hell. When to regenerate that file? With what compiler version and config? C/C++? With/out cleanup code? And where to put it? Are you sure that place will be writable for all compiler runs? And will the result be guaranteed to be reproducible, regardless of how many compiler runs there were in between? >>>>>> Unions don't really have a Python parallel, >>>>> >>>>> They can be a cdef class wrapping the union type. >>>> >>>> But I would think coercion would be difficult. Unions are usually (in >>>> my limited experience) for "don't worry about the type, just make sure >>>> it fits in X bytes". How would union->Python conversion work? >>> >>> There would be a wrapping type, e.g. >>> >>> cdef class MyUnion: >>> cdef union_type value Wouldn't that have to be a pointer to the real thing instead? >>> with a bunch of setters/getters for the values, just like there are >>> for structs. (In fact the same code would handle structs and unions). >>> >>> This is getting into the wrapper-generator territory, but I'm starting >>> to think for simple things that might be worth it. >> >> I think that if Cython will automatically generate a wrapper for >> >> cdef public int x >> >> it should generate a wrapper for >> >> cdef struct X: cdef public int x > > Or > > cdef public struct X: > int x > readonly int z > private int z > > I would perhaps say that non-Pythonable non-private members in public > structs would be a compile error. +1, keep it safe at the beginning. >> There really aren't that metatypes in C, so it doesn't seem like a >> slippery slope to me. Maybe I'm just missing something... >> >>>> Ok, I think we're pretty much agreed ;). I think that the next step >>>> is to start working on implementations of: >>>> >>>> * Stand alone .pxd -> Python module >>> >>> I'm not sure we're agreed on this one. Same from here. To me, that doesn't make much sense for code that wraps a library. And if it doesn't wrap a library, there isn't much benefit in writing a stand-alone .pxd in the first place. A .pyx is much more explicit and obvious in this case. Especially having some .pxd files that generate .so files and others that don't will make this very ugly. I'd prefer adding support for cimporting from .pyx files instead, potentially with an automated caching generation of corresponding .pxd files (maybe as ".pxdg" files to make them easier to handle for users). However, cyclic dependencies would be tricky to handle automatically then. >>>> * Extending class cdef/cdpef/public/readonly handling to cover enums, >>>> stucts, and possibly unions. >>> >>> This seems like the best first step. +1 >>>> * I don't know how to handle things like dummy enums (perhaps by >>>> requiring all cdef-ed enums to be named). >>> >>> All enums in C are named. >> >> But my Cython declaration (exposing a C `#define CONST_A 1`): >> >> cdef extern from 'mylib.h': >> enum: CONST_A >> >> is not a named enum. > > Ah, yes. Maybe we require a name (that would only be used in Python space). ... require it for cpdef enums, you mean? OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to declare external constants correctly. After all, we loose all type information that way. I just saw that in math.pxd things like "M_PI" are declared as plain "enum" instead of "const double" or something. The type inferencer can't do anything with that. It might even draw the completely wrong conclusions. Stefan From robertwb at math.washington.edu Sat Feb 19 19:10:08 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 19 Feb 2011 10:10:08 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D5F8C35.4010309@behnel.de> References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> <20110218043841.GA4195@tyr.home.net> <4D5F8C35.4010309@behnel.de> Message-ID: On Sat, Feb 19, 2011 at 1:24 AM, Stefan Behnel wrote: >>>>>>> A side effect of this cpdef change would be that now even bare .pxd >>>>>>> files (no matching .pyx) would have a Python presence, >>>>>> >>>>>> Where would it live? Would we just create this module (in essence, >>>>>> acting as if there was an empty .pyx file sitting there as well)? On >>>>>> this note, it may be worth pursuing the idea of a "cython helper" >>>>>> module where common code and objects could live. >>>>> >>>>> I'm not sure exactly what you mean by "cython helper", but this sounds >>>>> like my 'bare .pyx can create a Python .so module idea above. >>>> >>>> I'm thinking of a place to put, e.g. the generator and bind-able >>>> function classes, which are now re-implemented in every module that >>>> uses them. I think there will be more cases like this in the future >>>> rather than less. C-level code could be #included and linked from >>>> "global" stores as well. However, that's somewhat tangential. > > If you generate more than one file from a .pyx, including files that are > shared between compiler runs (or even readily built as .so files), you'd > quickly end up in dependency hell. When to regenerate that file? With what > compiler version and config? C/C++? With/out cleanup code? And where to put > it? Are you sure that place will be writable for all compiler runs? And will > the result be guaranteed to be reproducible, regardless of how many compiler > runs there were in between? I'd probably make it an option to the cythonize(...) command, so all the regeneration/dependency information would be computed based on the whole fileset rather than trying to figure that out for each individual module compilation. I'm thinking this would be an optimization, not a requirement. >>>>>>> Unions don't really have a Python parallel, >>>>>> >>>>>> They can be a cdef class wrapping the union type. >>>>> >>>>> But I would think coercion would be difficult. ?Unions are usually (in >>>>> my limited experience) for "don't worry about the type, just make sure >>>>> it fits in X bytes". ?How would union->Python conversion work? >>>> >>>> There would be a wrapping type, e.g. >>>> >>>> cdef class MyUnion: >>>> ? ?cdef union_type value > > Wouldn't that have to be a pointer to the real thing instead? Why? This would be a lot messier to get right, and structs and unions already have pass-by-value semantics. >>>> with a bunch of setters/getters for the values, just like there are >>>> for structs. (In fact the same code would handle structs and unions). >>>> >>>> This is getting into the wrapper-generator territory, but I'm starting >>>> to think for simple things that might be worth it. >>> >>> I think that if Cython will automatically generate a wrapper for >>> >>> ? ?cdef public int x >>> >>> it should generate a wrapper for >>> >>> ? ?cdef struct X: cdef public int x >> >> Or >> >> ? ? cdef public struct X: >> ? ? ? ? int x >> ? ? ? ? readonly int z >> ? ? ? ? private int z >> >> I would perhaps say that non-Pythonable non-private members in public >> structs would be a compile error. > > +1, keep it safe at the beginning. > > >>> There really aren't that metatypes in C, so it doesn't seem like a >>> slippery slope to me. ?Maybe I'm just missing something... >>> >>>>> Ok, I think we're pretty much agreed ;). ?I think that the next step >>>>> is to start working on implementations of: >>>>> >>>>> * Stand alone .pxd -> ?Python module >>>> >>>> I'm not sure we're agreed on this one. > > Same from here. To me, that doesn't make much sense for code that wraps a > library. And if it doesn't wrap a library, there isn't much benefit in > writing a stand-alone .pxd in the first place. A .pyx is much more explicit > and obvious in this case. Especially having some .pxd files that generate > .so files and others that don't will make this very ugly. > > I'd prefer adding support for cimporting from .pyx files instead, > potentially with an automated caching generation of corresponding .pxd files > (maybe as ".pxdg" files to make them easier to handle for users). However, > cyclic dependencies would be tricky to handle automatically then. We could put the in the new __pycache__. >>>>> * Extending class cdef/cdpef/public/readonly handling to cover enums, >>>>> ?stucts, and possibly unions. >>>> >>>> This seems like the best first step. > > +1 > > >>>>> * I don't know how to handle things like dummy enums (perhaps by >>>>> ?requiring all cdef-ed enums to be named). >>>> >>>> All enums in C are named. >>> >>> But my Cython declaration (exposing a C `#define CONST_A 1`): >>> >>> ? ?cdef extern from 'mylib.h': >>> ? ? ? ?enum: CONST_A >>> >>> is not a named enum. >> >> Ah, yes. Maybe we require a name (that would only be used in Python >> space). > > ... require it for cpdef enums, you mean? > > OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to > declare external constants correctly. After all, we loose all type > information that way. I just saw that in math.pxd things like "M_PI" are > declared as plain "enum" instead of "const double" or something. The type > inferencer can't do anything with that. It might even draw the completely > wrong conclusions. +1 Const support would be really nice to have, especially given how much more C++ uses it. - Robert From wking at drexel.edu Sat Feb 19 20:22:03 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 19 Feb 2011 14:22:03 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D5F8C35.4010309@behnel.de> Message-ID: <20110219192202.GA14142@tyr.home.net> On Fri, Feb 18, 2011 at 02:08:04PM -0800, Robert Bradshaw wrote: > On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote: > > On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: > >> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: > >>> On Thu, Feb 17, 2011 at 01:25:10PM -0800, Robert Bradshaw wrote: > >>>> On Thu, Feb 17, 2011 at 5:29 AM, W. Trevor King wrote: > >>>> > On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: > >>>> >> On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King wrote: > >>>> >> > What I'm missing is a way to bind the ModuleScope namespace to a name > >>>> >> > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, > >>>> >> > name)` will work in expose.pyx. > >>>> >> > >>>> >> You have also hit into the thorny issue that .pxd files are used for > >>>> >> many things. They may be pure C library declarations with no Python > >>>> >> module backing, they may be declarations of (externally implemented) > >>>> >> Python modules (such as numpy.pxd), or they may be declarations for > >>>> >> Cython-implemented modules. > >>>> >> > >>>> >> Here's another idea, what if extern blocks could contain cpdef > >>>> >> declarations, which would automatically generate a Python-level > >>>> >> wrappers for the declared members (if possible, otherwise an error)? > >>>> > > >>>> > Ah, this sounds good! Of the three .pxd roles you list above, > >>>> > external Python modules (e.g. numpy) and Cython-implemented modules > >>>> > (e.g. matched .pxd/.pyx) both already have a presence in Python-space. > >>>> > What's missing is a way to give (where possible) declarations of > >>>> > external C libraries a Python presence. cpdef fills this hole nicely, > >>>> > since its whole purpose is to expose Python interfaces to > >>>> > C-based elements. > >>>> > >>>> In the case of external Python modules, I'm not so sure we want to > >>>> monkey-patch our stuff in > >>> > >>> I don't think any of the changes we are suggesting would require > >>> changes to existing code, so .pxd-s with external implementations > >>> wouldn't be affected unless they brough the changes upon themselves. > >> > >> Say, in numpy.pxd, I have > >> > >> cdef extern from "...": > >> cpdef struct obscure_internal_struct: > >> ... > >> > >> Do we add an "obscure_internal_struct" onto the (global) numpy module? > >> What if it conflicts with a (runtime) name? This is the issue I'm > >> bringing up. > > > > Defining a cpdef *and* a non-matching external implementation should > > raise a compile-time error. I agree that there is a useful > > distinction between external-C-library and external-Python-module .pxd > > wrappers. Perhaps your matching blank .py or .pyx file could serve as > > a marker that the .pxd file should be inflated into its own full > > fledged python module. I'm not even sure how you would go about > > adding attributes to the numpy module. When/how would the > > Cython-created attributes get added? > > Yes, this is exactly the issue. Ah, I'm retracting my agreement on the external-C-library and external-Python-module .pxd wrappers. There is no difference in how their .pxd files should be treated, and I now agree that .pxd files should not generate .so modules unless they have a paried .py/.pyx file. > > If you try to override anything in a .so compiled module at runtime, > > you'd get the same kind of error you currently do trying to rebind a > > compiled class' method. > > That's the desired behavior for statically-bound globals, but > implementing it is not so trivial. It's currently implemented for classes. Are modules that different from classes? >>> import types >>> types.ModuleType.__class__.__mro__ (, ) So you've got your standard __setattr__ to override with an error-message generator. What is the implementation difficulty? I am also unclear about the distinction between cdef and cpdef (perhaps this should be a new thread?). cpdef means "I'm declaring something with C and Python interfaces. The Python interface is a thin wrapper which can be rebound to an object of any type, leaving the static C interface inaccessible from Python." cdef [private] means "I'm declaring something that only has a C interface." cdef public means "I'm declaring something with C and Python interfaces backed by C data. Python code can alter the C data." cdef readonly means "I'm declaring something with C and Python interfaces backed by C data. Python code cannot alter the C data." This seems to be broken in Cython at the module level, since I can rebind a cdef-ed class but not a cpdef-ed method: $ cat square.pyx cdef class A (object): cdef public int value cpdef square(self): return self.value**2 $ python -c 'import pyximport; pyximport.install(); import xx; a = xx.A(); a.value = 3; print a.square(); a.square = lambda self: self.value' Traceback (most recent call last): File "", line 3, in AttributeError: 'square.A' object attribute 'square' is read-only $ python -c 'import pyximport; pyximport.install(); import square; square.A = object; print square.A; a = square.A(); a.value = 3' Traceback (most recent call last): File "", line 2, in AttributeError: 'object' object has no attribute 'value' So the cdef-ed A currently has a rebindable Python interface (acting like a hypothetical cpdef-ed class), but it's square method is not rebindable (acting like a hypothetical `cdef readonly`-ed method). > >>>> (and where would we do it--on the first import of a cimporting > >>>> module?) > >>> > >>> Compilation is an issue. I think that .pxd files should be able to be > >>> cythoned directly, since then they Cython can build any wrappers they > >>> request. If the file has a matching .pyx file, cythoning either one > >>> should compile both together, since they'll produce a single Python > >>> .so module. > >> > >> ... > > > > Under the mantra "explicit is better than implicit", we could have > > users add something like > > > > cdef module "modname" > > > > to any .pxd files that should be inflated into Python modules. .pxd > > files without such a tag would receive the current treatment, error on > > any cpdef, etc. The drawback of this approach is that it makes Cython > > more complicated, but if both behaviors are reasonable, there's > > probably no getting around that. > > The other drawback is that it subverts the usual filename <-> module > name convention that one usually expects. I've been convinced that the `cimport .pyx file` route is a better way to go. However, the filename <-> module mapping is troublesome for backing externally-implemented Python modules (e.g. numpy). If you wanted to write a .pxd file backing numpy.random, how would you go about getting your module installed in Cython/Includes/numpy/random.pxd or another path that cython would successfully match with `cimport numpy.random`? On Sat, Feb 19, 2011 at 10:24:05AM +0100, Stefan Behnel wrote: > Robert Bradshaw, 18.02.2011 23:08: > > On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote: > >> On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: > >>> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: > >>>>>> A side effect of this cpdef change would be that now even bare .pxd > >>>>>> files (no matching .pyx) would have a Python presence, > >>>>> > >>>>> Where would it live? Would we just create this module (in essence, > >>>>> acting as if there was an empty .pyx file sitting there as well)? On > >>>>> this note, it may be worth pursuing the idea of a "cython helper" > >>>>> module where common code and objects could live. > >>>> > >>>> I'm not sure exactly what you mean by "cython helper", but this sounds > >>>> like my 'bare .pyx can create a Python .so module idea above. > >>> > >>> I'm thinking of a place to put, e.g. the generator and bind-able > >>> function classes, which are now re-implemented in every module that > >>> uses them. I think there will be more cases like this in the future > >>> rather than less. C-level code could be #included and linked from > >>> "global" stores as well. However, that's somewhat tangential. > > If you generate more than one file from a .pyx, including files that are > shared between compiler runs (or even readily built as .so files), you'd > quickly end up in dependency hell. I disagree here, but I like your cimportable .pyx better, so it doesn't matter ;). > >>>>>> Unions don't really have a Python parallel, > >>>>> > >>>>> They can be a cdef class wrapping the union type. > >>>> > >>>> But I would think coercion would be difficult. Unions are usually (in > >>>> my limited experience) for "don't worry about the type, just make sure > >>>> it fits in X bytes". How would union->Python conversion work? > >>> > >>> There would be a wrapping type, e.g. > >>> > >>> cdef class MyUnion: > >>> cdef union_type value > > Wouldn't that have to be a pointer to the real thing instead? Do you mean `cdef union_type *value`? Why would the above version not work? The union type has a well defined size and a number of well defined interpretations, so I don't see the problem. > >>> with a bunch of setters/getters for the values, just like there are > >>> for structs. (In fact the same code would handle structs and unions). > >>> > >>> This is getting into the wrapper-generator territory, but I'm starting > >>> to think for simple things that might be worth it. > >> > >> I think that if Cython will automatically generate a wrapper for > >> > >> cdef public int x > >> > >> it should generate a wrapper for > >> > >> cdef struct X: cdef public int x > > > > Or > > > > cdef public struct X: > > int x > > readonly int z > > private int z > > > > I would perhaps say that non-Pythonable non-private members in public > > structs would be a compile error. > > +1, keep it safe at the beginning. -1, keep the code clean and the interface consistent ;). I think the struct syntax should be identical to the class syntax, with the exception that you can't bind methods to structs. That's the only real difference between structs and classes, isn't it? If safety with a new feature is a concern, a warning like "EXPERIMENTAL FEATURE" in the associated docs and compiler output should be sufficient. > >> There really aren't that metatypes in C, so it doesn't seem like a > >> slippery slope to me. Maybe I'm just missing something... > >> > >>>> Ok, I think we're pretty much agreed ;). I think that the next step > >>>> is to start working on implementations of: > >>>> > >>>> * Stand alone .pxd -> Python module > >>> > >>> I'm not sure we're agreed on this one. > > Same from here. To me, that doesn't make much sense for code that wraps a > library. And if it doesn't wrap a library, there isn't much benefit in > writing a stand-alone .pxd in the first place. A .pyx is much more explicit > and obvious in this case. Especially having some .pxd files that generate > .so files and others that don't will make this very ugly. > > I'd prefer adding support for cimporting from .pyx files instead, > potentially with an automated caching generation of corresponding .pxd > files (maybe as ".pxdg" files to make them easier to handle for users). > However, cyclic dependencies would be tricky to handle automatically then. That's fine with me, since I just checked and you can do cdef extern from 'somelib.h': in .pyx files. Why would cyclic dependencies be difficult? The contents of .pxdg files should only depend on the paired .pyx file, so there is no need to parse another .pyx/.pxd file when generating a .pxdg. After you've created a .pxdg file, you're in the same situation that Cython presumably already handles well, with a .pyx/.pxd(g) pair. > >>>> * Extending class cdef/cdpef/public/readonly handling to cover enums, > >>>> stucts, and possibly unions. > >>> > >>> This seems like the best first step. > > +1 Working on it... > >>>> * I don't know how to handle things like dummy enums (perhaps by > >>>> requiring all cdef-ed enums to be named). > >>> > >>> All enums in C are named. > >> > >> But my Cython declaration (exposing a C `#define CONST_A 1`): > >> > >> cdef extern from 'mylib.h': > >> enum: CONST_A > >> > >> is not a named enum. > > > > Ah, yes. Maybe we require a name (that would only be used in Python space). > > ... require it for cpdef enums, you mean? > > OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to > declare external constants correctly. After all, we loose all type > information that way. I just saw that in math.pxd things like "M_PI" are > declared as plain "enum" instead of "const double" or something. The type > inferencer can't do anything with that. It might even draw the completely > wrong conclusions. Something like: [cdef|cpdef] extern [public|readonly] For example: cdef extern readonly double M_PI That would be nice, since the C compiler would (I think) raise an error when you try to use an invalid for macro value. [1]: http://projects.scipy.org/numpy/ticket/1686 -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From wking at drexel.edu Sat Feb 19 20:35:25 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 19 Feb 2011 14:35:25 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> Message-ID: <20110219193525.GB14142@tyr.home.net> On Thu, Feb 17, 2011 at 03:53:13PM -0800, Robert Bradshaw wrote: > On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: > > * Extending class cdef/cdpef/public/readonly handling to cover enums, > > stucts, and possibly unions. > > This seems like the best first step. > > > Problems with me getting started now: > > > > * I don't know where I should start mucking about in the source. > > ...The other files to look at would be Parsing.py,.. I've got the the Parsing pretty much working, but portions of the test suite are now failing with things like === Expected errors: === 1:5: function definition in pxd file must be declared 'cdef inline' 4:5: inline function definition in pxd file cannot be 'public' 7:5: inline function definition in pxd file cannot be 'api' === Got errors: === 1:5: function definition in pxd file must be declared 'cdef inline' 4:12: inline function definition in pxd file cannot be 'public' 7:5: inline function definition in pxd file cannot be 'api' while cythoning tests/errors/e_func_in_pxd_support.pxd: cdef foo(): return 1 cdef public inline foo2(): return 1 cdef api inline foo3(): return 1 The current Cython implementation does not consider 'cdef' to be part of the node code, and my current implementation does not consider any qualifiers to be part of the node code. It's unclear to me what the "right" position is for the start of a function (or variable, class, ...) should be, or how errors should be formatted. My initial impression is that Node.pos should point to the beginning of the def (here 1:1, 4:1, and 7:1), but that positions of the various qualifiers (cdef, public, inline, api, ...) should be cached so that the error points to the offending qualifier. Thoughs? -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sat Feb 19 21:47:41 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 19 Feb 2011 12:47:41 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110219192202.GA14142@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> Message-ID: On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: >> > If you try to override anything in a .so compiled module at runtime, >> > you'd get the same kind of error you currently do trying to rebind a >> > compiled class' method. >> >> That's the desired behavior for statically-bound globals, but >> implementing it is not so trivial. > > It's currently implemented for classes. ?Are modules that different > from classes? > > ? ?>>> import types > ? ?>>> types.ModuleType.__class__.__mro__ > ? ? ? ? ? ? ? ?(, ) > > So you've got your standard __setattr__ to override with an > error-message generator. ?What is the implementation difficulty? You can't just add a __setattr__ to a module and have it work, it's a class-level slot. And you don't want to modify all module classes. To do this you have to subclass module itself and insert that in place during import. > I am also unclear about the distinction between cdef and cpdef > (perhaps this should be a new thread?). > > cpdef means "I'm declaring something with C and Python interfaces. > ?The Python interface is a thin wrapper which can be rebound to an > ?object of any type, leaving the static C interface inaccessible from > ?Python." No, you can't re-bind cdef class methods. Despite Cython's attempt to homogenize things for the user, extension classes are quite different than "normal" Python classes. This is a Python C/API issue. > cdef [private] means "I'm declaring something that only has a C > ?interface." > cdef public means "I'm declaring something with C and Python > ?interfaces backed by C data. ?Python code can alter the C data." > cdef readonly means "I'm declaring something with C and Python > ?interfaces backed by C data. ?Python code cannot alter the C data." cdef means "back this by a C variable" > This seems to be broken in Cython at the module level, since I can > rebind a cdef-ed class but not a cpdef-ed method: Yes, we don't currently control the module's set/getattr. And classes are "public" by default. > ? ?$ cat square.pyx > ? ?cdef class A (object): > ? ? ? ?cdef public int value > > ? ? ? ?cpdef square(self): > ? ? ? ? ? ?return self.value**2 > ? ?$ python -c 'import pyximport; pyximport.install(); import xx; > ? ?a = xx.A(); a.value = 3; print a.square(); > ? ?a.square = lambda self: self.value' > ? ?Traceback (most recent call last): > ? ? ?File "", line 3, in > ? ?AttributeError: 'square.A' object attribute 'square' is read-only > ? ?$ python -c 'import pyximport; pyximport.install(); import square; > ? ?square.A = object; print square.A; a = square.A(); a.value = 3' > ? ? > ? ?Traceback (most recent call last): > ? ? ?File "", line 2, in > ? ?AttributeError: 'object' object has no attribute 'value' > > So the cdef-ed A currently has a rebindable Python interface (acting > like a hypothetical cpdef-ed class), but it's square method is not > rebindable (acting like a hypothetical `cdef readonly`-ed method). > >> >>>> (and where would we do it--on the first import of a cimporting >> >>>> module?) >> >>> >> >>> Compilation is an issue. ?I think that .pxd files should be able to be >> >>> cythoned directly, since then they Cython can build any wrappers they >> >>> request. ?If the file has a matching .pyx file, cythoning either one >> >>> should compile both together, since they'll produce a single Python >> >>> .so module. >> >> >> >> ... >> > >> > Under the mantra "explicit is better than implicit", we could have >> > users add something like >> > >> > ? ?cdef module "modname" >> > >> > to any .pxd files that should be inflated into Python modules. ?.pxd >> > files without such a tag would receive the current treatment, error on >> > any cpdef, etc. ?The drawback of this approach is that it makes Cython >> > more complicated, but if both behaviors are reasonable, there's >> > probably no getting around that. >> >> The other drawback is that it subverts the usual filename <-> module >> name convention that one usually expects. > > I've been convinced that the `cimport .pyx file` route is a better way > to go. > > However, the filename <-> module mapping is troublesome for backing > externally-implemented Python modules (e.g. numpy). ?If you wanted to > write a .pxd file backing numpy.random, how would you go about getting > your module installed in Cython/Includes/numpy/random.pxd or another > path that cython would successfully match with `cimport numpy.random`? Note that extern blocks (by definition) declare where things come from. > On Sat, Feb 19, 2011 at 10:24:05AM +0100, Stefan Behnel wrote: >> Robert Bradshaw, 18.02.2011 23:08: >> > On Thu, Feb 17, 2011 at 8:38 PM, W. Trevor King wrote: >> >> On Thu, Feb 17, 2011 at 3:53 PM, Robert Bradshaw wrote: >> >>> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >> >>>>>> A side effect of this cpdef change would be that now even bare .pxd >> >>>>>> files (no matching .pyx) would have a Python presence, >> >>>>> >> >>>>> Where would it live? Would we just create this module (in essence, >> >>>>> acting as if there was an empty .pyx file sitting there as well)? On >> >>>>> this note, it may be worth pursuing the idea of a "cython helper" >> >>>>> module where common code and objects could live. >> >>>> >> >>>> I'm not sure exactly what you mean by "cython helper", but this sounds >> >>>> like my 'bare .pyx can create a Python .so module idea above. >> >>> >> >>> I'm thinking of a place to put, e.g. the generator and bind-able >> >>> function classes, which are now re-implemented in every module that >> >>> uses them. I think there will be more cases like this in the future >> >>> rather than less. C-level code could be #included and linked from >> >>> "global" stores as well. However, that's somewhat tangential. >> >> If you generate more than one file from a .pyx, including files that are >> shared between compiler runs (or even readily built as .so files), you'd >> quickly end up in dependency hell. > > I disagree here, but I like your cimportable .pyx better, so it > doesn't matter ;). > >> >>>>>> Unions don't really have a Python parallel, >> >>>>> >> >>>>> They can be a cdef class wrapping the union type. >> >>>> >> >>>> But I would think coercion would be difficult. ?Unions are usually (in >> >>>> my limited experience) for "don't worry about the type, just make sure >> >>>> it fits in X bytes". ?How would union->Python conversion work? >> >>> >> >>> There would be a wrapping type, e.g. >> >>> >> >>> cdef class MyUnion: >> >>> ? ? cdef union_type value >> >> Wouldn't that have to be a pointer to the real thing instead? > > Do you mean `cdef union_type *value`? ?Why would the above version not > work? ?The union type has a well defined size and a number of well > defined interpretations, so I don't see the problem. > >> >>> with a bunch of setters/getters for the values, just like there are >> >>> for structs. (In fact the same code would handle structs and unions). >> >>> >> >>> This is getting into the wrapper-generator territory, but I'm starting >> >>> to think for simple things that might be worth it. >> >> >> >> I think that if Cython will automatically generate a wrapper for >> >> >> >> ? ? cdef public int x >> >> >> >> it should generate a wrapper for >> >> >> >> ? ? cdef struct X: cdef public int x >> > >> > Or >> > >> > ? ? ?cdef public struct X: >> > ? ? ? ? ?int x >> > ? ? ? ? ?readonly int z >> > ? ? ? ? ?private int z >> > >> > I would perhaps say that non-Pythonable non-private members in public >> > structs would be a compile error. >> >> +1, keep it safe at the beginning. > > -1, keep the code clean and the interface consistent ;). ?I think the > struct syntax should be identical to the class syntax, with the > exception that you can't bind methods to structs. ?That's the only > real difference between structs and classes, isn't it? In C++, the only difference between structs and classes is that struct members are public by default. (Not saying that C++ is always the model to follow, but it gives precedent). And structs can have function members, that's how to do OOP in C. > If safety with a new feature is a concern, a warning like > "EXPERIMENTAL FEATURE" in the associated docs and compiler output > should be sufficient. I think the point of "safe" is to start out with a compiler error, and we can change our minds later, which is better than trying to make legal statements illegal in the future. >> >> There really aren't that metatypes in C, so it doesn't seem like a >> >> slippery slope to me. ?Maybe I'm just missing something... >> >> >> >>>> Ok, I think we're pretty much agreed ;). ?I think that the next step >> >>>> is to start working on implementations of: >> >>>> >> >>>> * Stand alone .pxd -> ?Python module >> >>> >> >>> I'm not sure we're agreed on this one. >> >> Same from here. To me, that doesn't make much sense for code that wraps a >> library. And if it doesn't wrap a library, there isn't much benefit in >> writing a stand-alone .pxd in the first place. A .pyx is much more explicit >> and obvious in this case. Especially having some .pxd files that generate >> .so files and others that don't will make this very ugly. >> >> I'd prefer adding support for cimporting from .pyx files instead, >> potentially with an automated caching generation of corresponding .pxd >> files (maybe as ".pxdg" files to make them easier to handle for users). >> However, cyclic dependencies would be tricky to handle automatically then. > > That's fine with me, since I just checked and you can do > ? ?cdef extern from 'somelib.h': > in .pyx files. > > Why would cyclic dependencies be difficult? ?The contents of .pxdg > files should only depend on the paired .pyx file, so there is no need > to parse another .pyx/.pxd file when generating a .pxdg. ?After you've > created a .pxdg file, you're in the same situation that Cython > presumably already handles well, with a .pyx/.pxd(g) pair. Cyclic dependancies might require some special work, but they already do. >> >>>> * Extending class cdef/cdpef/public/readonly handling to cover enums, >> >>>> ? stucts, and possibly unions. >> >>> >> >>> This seems like the best first step. >> >> +1 > > Working on it... > >> >>>> * I don't know how to handle things like dummy enums (perhaps by >> >>>> ? requiring all cdef-ed enums to be named). >> >>> >> >>> All enums in C are named. >> >> >> >> But my Cython declaration (exposing a C `#define CONST_A 1`): >> >> >> >> ? ? cdef extern from 'mylib.h': >> >> ? ? ? ? enum: CONST_A >> >> >> >> is not a named enum. >> > >> > Ah, yes. Maybe we require a name (that would only be used in Python space). >> >> ... require it for cpdef enums, you mean? >> >> OTOH, the "enum: NAME" scheme is ugly by itself. There should be a way to >> declare external constants correctly. After all, we loose all type >> information that way. I just saw that in math.pxd things like "M_PI" are >> declared as plain "enum" instead of "const double" or something. The type >> inferencer can't do anything with that. It might even draw the completely >> wrong conclusions. > > Something like: > > ? ?[cdef|cpdef] extern [public|readonly] > > For example: > > ? ? ? ? ? ? ? ?cdef extern readonly double M_PI > > That would be nice, since the C compiler would (I think) raise an error > when you try to use an invalid for macro value. Const is different than readonly, as readonly specifies the python-level accessibility. - Robert From robertwb at math.washington.edu Sat Feb 19 21:52:38 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 19 Feb 2011 12:52:38 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110219193525.GB14142@tyr.home.net> References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> <20110219193525.GB14142@tyr.home.net> Message-ID: On Sat, Feb 19, 2011 at 11:35 AM, W. Trevor King wrote: > On Thu, Feb 17, 2011 at 03:53:13PM -0800, Robert Bradshaw wrote: >> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: >> > * Extending class cdef/cdpef/public/readonly handling to cover enums, >> > ?stucts, and possibly unions. >> >> This seems like the best first step. >> >> > Problems with me getting started now: >> > >> > * I don't know where I should start mucking about in the source. >> >> ...The other files to look at would be Parsing.py,.. > > I've got the the Parsing pretty much working, but portions of the test > suite are now failing with things like > > ? ?=== Expected errors: === > ? ?1:5: function definition in pxd file must be declared 'cdef inline' > ? ?4:5: inline function definition in pxd file cannot be 'public' > ? ?7:5: inline function definition in pxd file cannot be 'api' > > > ? ?=== Got errors: === > ? ?1:5: function definition in pxd file must be declared 'cdef inline' > ? ?4:12: inline function definition in pxd file cannot be 'public' > ? ?7:5: inline function definition in pxd file cannot be 'api' > > while cythoning tests/errors/e_func_in_pxd_support.pxd: > > ? ?cdef foo(): > ? ? ? ?return 1 > > ? ?cdef public inline foo2(): > ? ? ? ?return 1 > > ? ?cdef api inline foo3(): > ? ? ? ?return 1 > > The current Cython implementation does not consider 'cdef' to be part > of the node code, and my current implementation does not consider any > qualifiers to be part of the node code. > > It's unclear to me what the "right" position is for the start of a > function (or variable, class, ...) should be, or how errors should be > formatted. ?My initial impression is that Node.pos should point to the > beginning of the def (here 1:1, 4:1, and 7:1), but that positions of > the various qualifiers (cdef, public, inline, api, ...) should be > cached so that the error points to the offending qualifier. For better or for worse, Cython follows the same inane declarator rules as C, so cdef int a, *b, c[5], (*d)(int) is valid. This colors how c function declarations are expressed in the tree. Qualifiers don't really live in the tree and I'm not sure plumbing the "qualifier node" positions around would be worth the added complexity. - Robert From wking at drexel.edu Sat Feb 19 22:06:10 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 19 Feb 2011 16:06:10 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> <20110219193525.GB14142@tyr.home.net> Message-ID: <20110219210609.GA25132@tyr.home.net> On Sat, Feb 19, 2011 at 12:52:38PM -0800, Robert Bradshaw wrote: > On Sat, Feb 19, 2011 at 11:35 AM, W. Trevor King wrote: > > On Thu, Feb 17, 2011 at 03:53:13PM -0800, Robert Bradshaw wrote: > >> On Thu, Feb 17, 2011 at 3:12 PM, W. Trevor King wrote: > >> > * Extending class cdef/cdpef/public/readonly handling to cover enums, > >> > stucts, and possibly unions. > >> > >> This seems like the best first step. > >> > >> > Problems with me getting started now: > >> > > >> > * I don't know where I should start mucking about in the source. > >> > >> ...The other files to look at would be Parsing.py,.. > > > > I've got the the Parsing pretty much working, but portions of the test > > suite are now failing with things like > > > > === Expected errors: === > > 1:5: function definition in pxd file must be declared 'cdef inline' > > 4:5: inline function definition in pxd file cannot be 'public' > > 7:5: inline function definition in pxd file cannot be 'api' > > > > > > === Got errors: === > > 1:5: function definition in pxd file must be declared 'cdef inline' > > 4:12: inline function definition in pxd file cannot be 'public' > > 7:5: inline function definition in pxd file cannot be 'api' > > > > while cythoning tests/errors/e_func_in_pxd_support.pxd: > > > > cdef foo(): > > return 1 > > > > cdef public inline foo2(): > > return 1 > > > > cdef api inline foo3(): > > return 1 > > > > The current Cython implementation does not consider 'cdef' to be part > > of the node code, and my current implementation does not consider any > > qualifiers to be part of the node code. > > > > It's unclear to me what the "right" position is for the start of a > > function (or variable, class, ...) should be, or how errors should be > > formatted. My initial impression is that Node.pos should point to the > > beginning of the def (here 1:1, 4:1, and 7:1), but that positions of > > the various qualifiers (cdef, public, inline, api, ...) should be > > cached so that the error points to the offending qualifier. > > For better or for worse, Cython follows the same inane declarator > rules as C, so > > cdef int a, *b, c[5], (*d)(int) > > is valid. This colors how c function declarations are expressed in the > tree. Qualifiers don't really live in the tree and I'm not sure > plumbing the "qualifier node" positions around would be worth the > added complexity. So should I go ahead and update the expected error messages in my branch? -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sat Feb 19 22:33:53 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 19 Feb 2011 13:33:53 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110219210609.GA25132@tyr.home.net> References: <20110209172325.GA4661@tyr.home.net> <20110216161721.GA16736@tyr.home.net> <20110217132940.GA28686@tyr.home.net> <20110217231235.GA641@tyr.home.net> <20110219193525.GB14142@tyr.home.net> <20110219210609.GA25132@tyr.home.net> Message-ID: On Sat, Feb 19, 2011 at 1:06 PM, W. Trevor King wrote: > So should I go ahead and update the expected error messages in my branch? Yes. From wking at drexel.edu Sat Feb 19 22:45:16 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 19 Feb 2011 16:45:16 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> Message-ID: <20110219214515.GB25132@tyr.home.net> On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: > On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: > > >> > If you try to override anything in a .so compiled module at runtime, > >> > you'd get the same kind of error you currently do trying to rebind a > >> > compiled class' method. > >> > >> That's the desired behavior for statically-bound globals, but > >> implementing it is not so trivial. > > > > It's currently implemented for classes. Are modules that different > > from classes? > > > > >>> import types > > >>> types.ModuleType.__class__.__mro__ > > (, ) > > > > So you've got your standard __setattr__ to override with an > > error-message generator. What is the implementation difficulty? > > You can't just add a __setattr__ to a module and have it work, it's a > class-level slot. And you don't want to modify all module classes. To > do this you have to subclass module itself and insert that in place > during import. So annoying but possible? Not particularly critical either way, though, since you could always say "don't rebind module-level stuff" in a project's docs, even if you don't enforce that at the Cython level. > > I am also unclear about the distinction between cdef and cpdef > > (perhaps this should be a new thread?). > > > > cpdef means "I'm declaring something with C and Python interfaces. > > The Python interface is a thin wrapper which can be rebound to an > > object of any type, leaving the static C interface inaccessible from > > Python." > > No, you can't re-bind cdef class methods. Despite Cython's attempt to > homogenize things for the user, extension classes are quite different > than "normal" Python classes. This is a Python C/API issue. Do you mean `cpdef class methods`? If so, you're right: $ cat square.pyx cdef class A (object): cdef public int value cpdef square(self): return self.value**2 $ python -c 'import pyximport; pyximport.install(); import square; square.A.square = lambda self: self.value' Traceback (most recent call last): File "", line 1, in TypeError: can't set attributes of built-in/extension type 'square.A' You can't override them in instances either: $ python -c 'import pyximport; pyximport.install(); import square; a = square.A(); a.square = lambda self: self.value' Traceback (most recent call last): File "", line 1, in AttributeError: 'square.A' object attribute 'square' is read-only But you can override them in subclasses, which is, I suppose, the point of cpdef for methods. > > cdef [private] means "I'm declaring something that only has a C > > interface." > > cdef public means "I'm declaring something with C and Python > > interfaces backed by C data. Python code can alter the C data." > > cdef readonly means "I'm declaring something with C and Python > > interfaces backed by C data. Python code cannot alter the C data." > > cdef means "back this by a C variable" Ah, ok. That makes more sense. > > However, the filename <-> module mapping is troublesome for backing > > externally-implemented Python modules (e.g. numpy). If you wanted to > > write a .pxd file backing numpy.random, how would you go about getting > > your module installed in Cython/Includes/numpy/random.pxd or another > > path that cython would successfully match with `cimport numpy.random`? > > Note that extern blocks (by definition) declare where things come from. They declare where the .pxd file looks for .h files, but not where .pyx files look for the .pxd file. > >> > cdef public struct X: > >> > int x > >> > readonly int z > >> > private int z > >> > > >> > I would perhaps say that non-Pythonable non-private members in public > >> > structs would be a compile error. > >> > >> +1, keep it safe at the beginning. > > > > -1, keep the code clean and the interface consistent ;). I think the > > struct syntax should be identical to the class syntax, with the > > exception that you can't bind methods to structs. That's the only > > real difference between structs and classes, isn't it? > > In C++, the only difference between structs and classes is that struct > members are public by default. (Not saying that C++ is always the > model to follow, but it gives precedent). And structs can have > function members, that's how to do OOP in C. Oh. Even more reason to have identical struct and class handling in Cython ;). It is unclear to me what `cdef public struct` means. I think it should mean "Python bindings can alter this struct's definition", which doesn't make sense. Shouldn't the syntax for public members be cdef struct X: cdef public: int x readonly int y private int z > > If safety with a new feature is a concern, a warning like > > "EXPERIMENTAL FEATURE" in the associated docs and compiler output > > should be sufficient. > > I think the point of "safe" is to start out with a compiler error, and > we can change our minds later, which is better than trying to make > legal statements illegal in the future. Ok, but I still don't understand why the cdefs were removed from the proposed structure members, when they are required for class definitions. > > That would be nice, since the C compiler would (I think) raise an error > > when you try to use an invalid for macro value. > > Const is different than readonly, as readonly specifies the > python-level accessibility. Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to consolidate the way these are handled in Parsing/Nodes/Symtab and I want to make sure I don't implement the wrong interpretation. Can you clarify how one knows if "public" means "expose a read/write Python interface to this object" or "expose this symbol to external C code"? -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sat Feb 19 23:04:16 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 19 Feb 2011 14:04:16 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110219214515.GB25132@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> Message-ID: On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: >> >> >> > If you try to override anything in a .so compiled module at runtime, >> >> > you'd get the same kind of error you currently do trying to rebind a >> >> > compiled class' method. >> >> >> >> That's the desired behavior for statically-bound globals, but >> >> implementing it is not so trivial. >> > >> > It's currently implemented for classes. ?Are modules that different >> > from classes? >> > >> > ? ?>>> import types >> > ? ?>>> types.ModuleType.__class__.__mro__ >> > ? ? ? ? ? ? ? ?(, ) >> > >> > So you've got your standard __setattr__ to override with an >> > error-message generator. ?What is the implementation difficulty? >> >> You can't just add a __setattr__ to a module and have it work, it's a >> class-level slot. And you don't want to modify all module classes. To >> do this you have to subclass module itself and insert that in place >> during import. > > So annoying but possible? ?Not particularly critical either way, > though, since you could always say "don't rebind module-level stuff" > in a project's docs, even if you don't enforce that at the Cython > level. It's still something I'd like to do. This could also be used to allow fast access to module globals and easier sharing of declarations. >> > I am also unclear about the distinction between cdef and cpdef >> > (perhaps this should be a new thread?). >> > >> > cpdef means "I'm declaring something with C and Python interfaces. >> > ?The Python interface is a thin wrapper which can be rebound to an >> > ?object of any type, leaving the static C interface inaccessible from >> > ?Python." >> >> No, you can't re-bind cdef class methods. Despite Cython's attempt to >> homogenize things for the user, extension classes are quite different >> than "normal" Python classes. This is a Python C/API issue. > > Do you mean `cpdef class methods`? ?If so, you're right: > > ? ?$ cat square.pyx > ? ?cdef class A (object): > ? ? ? ?cdef public int value > > ? ? ? ?cpdef square(self): > ? ? ? ? ? ?return self.value**2 > ? ?$ python -c 'import pyximport; pyximport.install(); import square; > ? ? ? ? ? ? ? ?square.A.square = lambda self: self.value' > ? ?Traceback (most recent call last): > ? ? ?File "", line 1, in > ? ?TypeError: can't set attributes of built-in/extension type 'square.A' > > You can't override them in instances either: > > ? ?$ python -c 'import pyximport; pyximport.install(); import square; > ? ?a = square.A(); a.square = lambda self: self.value' > ? ?Traceback (most recent call last): > ? ? ?File "", line 1, in > ? ?AttributeError: 'square.A' object attribute 'square' is read-only > > But you can override them in subclasses, which is, I suppose, the > point of cpdef for methods. Yes, otherwise they would have been pretty easy to implement :). >> > cdef [private] means "I'm declaring something that only has a C >> > ?interface." >> > cdef public means "I'm declaring something with C and Python >> > ?interfaces backed by C data. ?Python code can alter the C data." >> > cdef readonly means "I'm declaring something with C and Python >> > ?interfaces backed by C data. ?Python code cannot alter the C data." >> >> cdef means "back this by a C variable" > > Ah, ok. ?That makes more sense. > >> > However, the filename <-> module mapping is troublesome for backing >> > externally-implemented Python modules (e.g. numpy). ?If you wanted to >> > write a .pxd file backing numpy.random, how would you go about getting >> > your module installed in Cython/Includes/numpy/random.pxd or another >> > path that cython would successfully match with `cimport numpy.random`? >> >> Note that extern blocks (by definition) declare where things come from. > > They declare where the .pxd file looks for .h files, but not where > .pyx files look for the .pxd file. Sorry, I should have said extern blocks that make cdef class declarations (such as our numpy.pxd). >> >> > ? ? ?cdef public struct X: >> >> > ? ? ? ? ?int x >> >> > ? ? ? ? ?readonly int z >> >> > ? ? ? ? ?private int z >> >> > >> >> > I would perhaps say that non-Pythonable non-private members in public >> >> > structs would be a compile error. >> >> >> >> +1, keep it safe at the beginning. >> > >> > -1, keep the code clean and the interface consistent ;). ?I think the >> > struct syntax should be identical to the class syntax, with the >> > exception that you can't bind methods to structs. ?That's the only >> > real difference between structs and classes, isn't it? >> >> In C++, the only difference between structs and classes is that struct >> members are public by default. (Not saying that C++ is always the >> model to follow, but it gives precedent). And structs can have >> function members, that's how to do OOP in C. > > Oh. ?Even more reason to have identical struct and class handling in > Cython ;). > > It is unclear to me what `cdef public struct` means. ?I think it > should mean "Python bindings can alter this struct's definition", > which doesn't make sense. I think it should mean "this struct is accessible from Python (as X)" > Shouldn't the syntax for public members be > > ? ?cdef struct X: > ? ? ? ?cdef public: > ? ? ? ? ? ?int x > ? ? ? ? ? ?readonly int y > ? ? ? ? ? ?private int z -1 on nesting things like this. Rather than make a struct visible from Python iff any of its members are, I think it makes more sense to put the declaration on the struct itself. We could support cdef public struct X: int x # public cdef readonly struct Y: int y # readonly cdef [private] struct Z: int z # private, as we don't even have Z in the Python namespace, and no wrapper is created. >> > If safety with a new feature is a concern, a warning like >> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output >> > should be sufficient. >> >> I think the point of "safe" is to start out with a compiler error, and >> we can change our minds later, which is better than trying to make >> legal statements illegal in the future. > > Ok, but I still don't understand why the cdefs were removed from the > proposed structure members, when they are required for class > definitions. Because structs can only have c members, so the cdef was entirely redundant. They weren't really removed per say, it's just that with the exception of cdef classes, "cdef ..." meant "a c declaration follows." >> > That would be nice, since the C compiler would (I think) raise an error >> > when you try to use an invalid for macro value. >> >> Const is different than readonly, as readonly specifies the >> python-level accessibility. > > Ah. ?Sorry for all the c(p)def/qualifier confusion, but I'm trying to > consolidate the way these are handled in Parsing/Nodes/Symtab and I > want to make sure I don't implement the wrong interpretation. ?Can you > clarify how one knows if "public" means "expose a read/write Python > interface to this object" or "expose this symbol to external C code"? Public has had several different meanings. I wish there were a spec and full grammer for Cython, but there's not (yet?). The meaning is implicit in the code, and there's enough users out there that we should stay backwards compatible. It may be worth doing some backwards-incompatible normalization before we hit 1.0, but compiling the entire Python grammar is higher priority than that. - Robert From wking at drexel.edu Sun Feb 20 00:31:27 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 19 Feb 2011 18:31:27 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> Message-ID: <20110219233126.GC25132@tyr.home.net> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: > On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: > > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: > >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: > >> > However, the filename <-> module mapping is troublesome for backing > >> > externally-implemented Python modules (e.g. numpy). If you wanted to > >> > write a .pxd file backing numpy.random, how would you go about getting > >> > your module installed in Cython/Includes/numpy/random.pxd or another > >> > path that cython would successfully match with `cimport numpy.random`? > >> > >> Note that extern blocks (by definition) declare where things come from. > > > > They declare where the .pxd file looks for .h files, but not where > > .pyx files look for the .pxd file. > > Sorry, I should have said extern blocks that make cdef class > declarations (such as our numpy.pxd). It doesn't look like there are cdef class declarations in numpy.pxd: cython $ grep class Cython/Includes/numpy.pxd ctypedef class numpy.dtype [object PyArray_Descr]: ctypedef extern class numpy.flatiter [object PyArrayIterObject]: ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: ctypedef class numpy.ndarray [object PyArrayObject]: ctypedef extern class numpy.ufunc [object PyUFuncObject]: This still doesn't explain how .pxd files specify which external implemented Python modules they correspond to. > >> >> > cdef public struct X: > >> >> > int x > >> >> > readonly int z > >> >> > private int z > >> >> > > >> >> > I would perhaps say that non-Pythonable non-private members in public > >> >> > structs would be a compile error. > >> >> > >> >> +1, keep it safe at the beginning. > >> > > >> > -1, keep the code clean and the interface consistent ;). I think the > >> > struct syntax should be identical to the class syntax, with the > >> > exception that you can't bind methods to structs. That's the only > >> > real difference between structs and classes, isn't it? > >> > >> In C++, the only difference between structs and classes is that struct > >> members are public by default. (Not saying that C++ is always the > >> model to follow, but it gives precedent). And structs can have > >> function members, that's how to do OOP in C. > > > > Oh. Even more reason to have identical struct and class handling in > > Cython ;). > > > > It is unclear to me what `cdef public struct` means. I think it > > should mean "Python bindings can alter this struct's definition", > > which doesn't make sense. > > I think it should mean "this struct is accessible from Python (as X)" Wouldn't that be "cdef readonly struct X"? > > Shouldn't the syntax for public members be > > > > cdef struct X: > > cdef public: > > int x > > readonly int y > > private int z > > -1 on nesting things like this. Rather than make a struct visible from > Python iff any of its members are, A struct is visible from python iff it is declared public or readonly: cdef public struct X: ... or cdef readonly struct X: ... I don't think the visibility of the struct as a whole should have any effect over the visibility of its members, so you should be able to specify member visibility explicitly with per-member granularity (as you currently can for classes). I was assuming that structs would be public by default (like classes), but that is obviously configurable. > I think it makes more sense to put > the declaration on the struct itself. We could support > > cdef public struct X: > int x # public > > cdef readonly struct Y: > int y # readonly > > cdef [private] struct Z: > int z # private, as we don't even have Z in the Python namespace, > and no wrapper is created. The problems with this are: * It's differnent from how we handle the almost identical class case. * It makes it impossible to define, for example a public struct with C-only attributes: cdef public struct X: cdef public int a cdef private void* ptr Obviously, public attributes of private structs should raise compile-time Cython errors. > >> > If safety with a new feature is a concern, a warning like > >> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output > >> > should be sufficient. > >> > >> I think the point of "safe" is to start out with a compiler error, and > >> we can change our minds later, which is better than trying to make > >> legal statements illegal in the future. > > > > Ok, but I still don't understand why the cdefs were removed from the > > proposed structure members, when they are required for class > > definitions. > > Because structs can only have c members, so the cdef was entirely > redundant. They weren't really removed per say, it's just that with > the exception of cdef classes, "cdef ..." meant "a c declaration > follows." But cdef classes can also only have cdef members. I think it's better to keep cdef meaning "backed by C data", not necessarily "written using C syntax", since you're trying to do more with Cython, so it doesn't make sense to force C syntax. > >> > That would be nice, since the C compiler would (I think) raise an error > >> > when you try to use an invalid for macro value. > >> > >> Const is different than readonly, as readonly specifies the > >> python-level accessibility. > > > > Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to > > consolidate the way these are handled in Parsing/Nodes/Symtab and I > > want to make sure I don't implement the wrong interpretation. Can you > > clarify how one knows if "public" means "expose a read/write Python > > interface to this object" or "expose this symbol to external C code"? > > Public has had several different meanings. I wish there were a spec > and full grammer for Cython, but there's not (yet?). The meaning is > implicit in the code, and there's enough users out there that we > should stay backwards compatible. It may be worth doing some > backwards-incompatible normalization before we hit 1.0, but compiling > the entire Python grammar is higher priority than that. Since I'm going to have lots of similar stuff (classes, enums, structs, unions) all with the same (hopefully) cdef/cpdef/visibility stuff for members, I'd like to consolidate now. I will of course, add special-case code as necessary to support the current syntax, which can then be removed whenever you think it is appropriate, but writing separate, near-identical handlers for each type seems like a recipe for disaster ;). I'll look to the code for guidance on public, and try to work out the appropriate meaning during the parse phase. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sun Feb 20 01:41:27 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 19 Feb 2011 16:41:27 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110219233126.GC25132@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> Message-ID: On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >> > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: >> >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: >> >> > However, the filename <-> module mapping is troublesome for backing >> >> > externally-implemented Python modules (e.g. numpy). ?If you wanted to >> >> > write a .pxd file backing numpy.random, how would you go about getting >> >> > your module installed in Cython/Includes/numpy/random.pxd or another >> >> > path that cython would successfully match with `cimport numpy.random`? >> >> >> >> Note that extern blocks (by definition) declare where things come from. >> > >> > They declare where the .pxd file looks for .h files, but not where >> > .pyx files look for the .pxd file. >> >> Sorry, I should have said extern blocks that make cdef class >> declarations (such as our numpy.pxd). > > It doesn't look like there are cdef class declarations in numpy.pxd: > > ? ?cython $ grep class Cython/Includes/numpy.pxd > ? ? ? ?ctypedef class numpy.dtype [object PyArray_Descr]: > ? ? ? ?ctypedef extern class numpy.flatiter [object PyArrayIterObject]: > ? ? ? ?ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: > ? ? ? ?ctypedef class numpy.ndarray [object PyArrayObject]: > ? ? ? ?ctypedef extern class numpy.ufunc [object PyUFuncObject]: > > This still doesn't explain how .pxd files specify which external > implemented Python modules they correspond to. "numpy.dtype" is the fully qualified name of the class it's declaring--the module is "numpy." >> >> >> > ? ? ?cdef public struct X: >> >> >> > ? ? ? ? ?int x >> >> >> > ? ? ? ? ?readonly int z >> >> >> > ? ? ? ? ?private int z >> >> >> > >> >> >> > I would perhaps say that non-Pythonable non-private members in public >> >> >> > structs would be a compile error. >> >> >> >> >> >> +1, keep it safe at the beginning. >> >> > >> >> > -1, keep the code clean and the interface consistent ;). ?I think the >> >> > struct syntax should be identical to the class syntax, with the >> >> > exception that you can't bind methods to structs. ?That's the only >> >> > real difference between structs and classes, isn't it? >> >> >> >> In C++, the only difference between structs and classes is that struct >> >> members are public by default. (Not saying that C++ is always the >> >> model to follow, but it gives precedent). And structs can have >> >> function members, that's how to do OOP in C. >> > >> > Oh. ?Even more reason to have identical struct and class handling in >> > Cython ;). >> > >> > It is unclear to me what `cdef public struct` means. ?I think it >> > should mean "Python bindings can alter this struct's definition", >> > which doesn't make sense. >> >> I think it should mean "this struct is accessible from Python (as X)" > > Wouldn't that be "cdef readonly struct X"? > >> > Shouldn't the syntax for public members be >> > >> > ? ?cdef struct X: >> > ? ? ? ?cdef public: >> > ? ? ? ? ? ?int x >> > ? ? ? ? ? ?readonly int y >> > ? ? ? ? ? ?private int z >> >> -1 on nesting things like this. Rather than make a struct visible from >> Python iff any of its members are, > > A struct is visible from python iff it is declared public or readonly: > > ? ?cdef public struct X: > ? ? ? ?... > > or > > ? ?cdef readonly struct X: > ? ? ? ?... > > I don't think the visibility of the struct as a whole should have any > effect over the visibility of its members, so you should be able to > specify member visibility explicitly with per-member granularity (as > you currently can for classes). > > I was assuming that structs would be public by default (like classes), > but that is obviously configurable. > >> I think it makes more sense to put >> the declaration on the struct itself. We could support >> >> cdef public struct X: >> ? ? int x # public >> >> cdef readonly struct Y: >> ? ? int y # readonly >> >> cdef [private] struct Z: >> ? ? int z # private, as we don't even have Z in the Python namespace, >> and no wrapper is created. > > The problems with this are: > > * It's differnent from how we handle the almost identical class case. True it's different, but there are significant differences between classes and structs in Cython, and this is the way things are now. > * It makes it impossible to define, for example a public struct with > ?C-only attributes: > > ? ?cdef public struct X: > ? ? ? ?cdef public int a > ? ? ? ?cdef private void* ptr ? The above would work just fine. I was proposing that it would be semantically equivalent to ? ?cdef public struct X: ? ? ? ?cdef int a ? ? ? ?cdef private void* ptr Perhaps that was not clear. Consider the currently valid cdef struct X: int a void* ptr We can't make everything public by default, as this would break valid code. (I don't think it's a good idea to make the default visibility of a member be a function of its type if we can help it.) The original proposal was to allow cpdef struct X: ... but making all fields private by default would be a bit useless, which was why I was suggesting they be public. We could allow the the modifier "cpdef readonly struct X" which would be the (probably less common) case where all members were by default readonly, though exposed to Python, rather than public. Within an exposed struct, of course, any member could be declared as private/readonly/public individually. BTW, the "public" keyword is the wrong thing to use here, as that actually controls name mangling and (c-level) symbol exporting. The fact that means a different thing for members than for top-level symbols isn't ideal, but at least it's unambiguous as members need not be mangled. > Obviously, public attributes of private structs should raise > compile-time Cython errors. > >> >> > If safety with a new feature is a concern, a warning like >> >> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output >> >> > should be sufficient. >> >> >> >> I think the point of "safe" is to start out with a compiler error, and >> >> we can change our minds later, which is better than trying to make >> >> legal statements illegal in the future. >> > >> > Ok, but I still don't understand why the cdefs were removed from the >> > proposed structure members, when they are required for class >> > definitions. >> >> Because structs can only have c members, so the cdef was entirely >> redundant. They weren't really removed per say, it's just that with >> the exception of cdef classes, "cdef ..." meant "a c declaration >> follows." > > But cdef classes can also only have cdef members. > > I think it's better to keep cdef meaning "backed by C data", not > necessarily "written using C syntax", since you're trying to do more > with Cython, so it doesn't make sense to force C syntax. It means both. Were I to start over, I would make "cdef int* a, b, c" declare three pointers, but we're stuck with the C syntax we have. In a struct "cdef" on members is entirely redundant (though I am not strongly opposed to allowing it). >> >> > That would be nice, since the C compiler would (I think) raise an error >> >> > when you try to use an invalid for macro value. >> >> >> >> Const is different than readonly, as readonly specifies the >> >> python-level accessibility. >> > >> > Ah. ?Sorry for all the c(p)def/qualifier confusion, but I'm trying to >> > consolidate the way these are handled in Parsing/Nodes/Symtab and I >> > want to make sure I don't implement the wrong interpretation. ?Can you >> > clarify how one knows if "public" means "expose a read/write Python >> > interface to this object" or "expose this symbol to external C code"? >> >> Public has had several different meanings. I wish there were a spec >> and full grammer for Cython, but there's not (yet?). The meaning is >> implicit in the code, and there's enough users out there that we >> should stay backwards compatible. It may be worth doing some >> backwards-incompatible normalization before we hit 1.0, but compiling >> the entire Python grammar is higher priority than that. > > Since I'm going to have lots of similar stuff (classes, enums, > structs, unions) all with the same (hopefully) cdef/cpdef/visibility > stuff for members, I'd like to consolidate now. ?I will of course, add > special-case code as necessary to support the current syntax, which > can then be removed whenever you think it is appropriate, but writing > separate, near-identical handlers for each type seems like a recipe > for disaster ;). Yes. For structs/unions, it's already almost always a StructOrUnion object anyways. Classes are somewhat special. > I'll look to the code for guidance on public, and try to work out the > appropriate meaning during the parse phase. Sounds good. - Robert From wking at drexel.edu Sun Feb 20 03:32:08 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 19 Feb 2011 21:32:08 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> Message-ID: <20110220023207.GA25084@tyr.home.net> On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote: > On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote: > > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: > >> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: > >> > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: > >> >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: > >> >> > However, the filename <-> module mapping is troublesome for backing > >> >> > externally-implemented Python modules (e.g. numpy). If you wanted to > >> >> > write a .pxd file backing numpy.random, how would you go about getting > >> >> > your module installed in Cython/Includes/numpy/random.pxd or another > >> >> > path that cython would successfully match with `cimport numpy.random`? > >> >> > >> >> Note that extern blocks (by definition) declare where things come from. > >> > > >> > They declare where the .pxd file looks for .h files, but not where > >> > .pyx files look for the .pxd file. > >> > >> Sorry, I should have said extern blocks that make cdef class > >> declarations (such as our numpy.pxd). > > > > It doesn't look like there are cdef class declarations in numpy.pxd: > > > > cython $ grep class Cython/Includes/numpy.pxd > > ctypedef class numpy.dtype [object PyArray_Descr]: > > ctypedef extern class numpy.flatiter [object PyArrayIterObject]: > > ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: > > ctypedef class numpy.ndarray [object PyArrayObject]: > > ctypedef extern class numpy.ufunc [object PyUFuncObject]: > > > > This still doesn't explain how .pxd files specify which external > > implemented Python modules they correspond to. > > "numpy.dtype" is the fully qualified name of the class it's > declaring--the module is "numpy." Hmm, that means that cimport numpy a = numpy.numpy.ndarray also compiles. Wouldn't it make more sense to mirror numpy's module structure when backing it? You do have nested .pxd modules for cpython, libc, etc. > >> >> >> > cdef public struct X: > >> >> >> > int x > >> >> >> > readonly int z > >> >> >> > private int z > >> >> >> > > >> >> >> > I would perhaps say that non-Pythonable non-private members in public > >> >> >> > structs would be a compile error. > >> >> >> > >> >> >> +1, keep it safe at the beginning. > >> >> > > >> >> > -1, keep the code clean and the interface consistent ;). I think the > >> >> > struct syntax should be identical to the class syntax, with the > >> >> > exception that you can't bind methods to structs. That's the only > >> >> > real difference between structs and classes, isn't it? > >> >> > >> >> In C++, the only difference between structs and classes is that struct > >> >> members are public by default. (Not saying that C++ is always the > >> >> model to follow, but it gives precedent). And structs can have > >> >> function members, that's how to do OOP in C. > >> > > >> > Oh. Even more reason to have identical struct and class handling in > >> > Cython ;). > >> > > >> > It is unclear to me what `cdef public struct` means. I think it > >> > should mean "Python bindings can alter this struct's definition", > >> > which doesn't make sense. > >> > >> I think it should mean "this struct is accessible from Python (as X)" > > > > Wouldn't that be "cdef readonly struct X"? > > > >> > Shouldn't the syntax for public members be > >> > > >> > cdef struct X: > >> > cdef public: > >> > int x > >> > readonly int y > >> > private int z > >> > >> -1 on nesting things like this. Rather than make a struct visible from > >> Python iff any of its members are, > > > > A struct is visible from python iff it is declared public or readonly: > > > > cdef public struct X: > > ... > > > > or > > > > cdef readonly struct X: > > ... > > > > I don't think the visibility of the struct as a whole should have any > > effect over the visibility of its members, so you should be able to > > specify member visibility explicitly with per-member granularity (as > > you currently can for classes). > > > > I was assuming that structs would be public by default (like classes), > > but that is obviously configurable. > > > >> I think it makes more sense to put > >> the declaration on the struct itself. We could support > >> > >> cdef public struct X: > >> int x # public > >> > >> cdef readonly struct Y: > >> int y # readonly > >> > >> cdef [private] struct Z: > >> int z # private, as we don't even have Z in the Python namespace, > >> and no wrapper is created. > > > > The problems with this are: > > > > * It's differnent from how we handle the almost identical class case. > > True it's different, but there are significant differences between > classes and structs in Cython, and this is the way things are now. Ah, well. I suppose I'll leave the Parser pretty much alone. Classes and structs will end up being much closer on the Python interface side, perhaps the syntax will grow closer in future Cythons... > > * It makes it impossible to define, for example a public struct with > > C-only attributes: > > > > cdef public struct X: > > cdef public int a > > cdef private void* ptr > > ? > > The above would work just fine. I was proposing that it would be > semantically equivalent to > > cdef public struct X: > cdef int a > cdef private void* ptr > > Perhaps that was not clear. Consider the currently valid > > cdef struct X: > int a > void* ptr > > We can't make everything public by default, as this would break valid > code. (I don't think it's a good idea to make the default visibility > of a member be a function of its type if we can help it.) Oh, I though (for some reason) that you were ruling out member-level visibility adjustments. Using the struct visibility to set member-level defaults is fine. > BTW, the "public" keyword is the wrong thing to use here, as that > actually controls name mangling and (c-level) symbol exporting. The > fact that means a different thing for members than for top-level > symbols isn't ideal, but at least it's unambiguous as members need not > be mangled. "public" means "read/write Python interface" for struct attributes, I was assuming that would also apply to struct members acording to tutorial/cdef_classes. Should I use a different visibility name in structs and unions? > >> >> > If safety with a new feature is a concern, a warning like > >> >> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output > >> >> > should be sufficient. > >> >> > >> >> I think the point of "safe" is to start out with a compiler error, and > >> >> we can change our minds later, which is better than trying to make > >> >> legal statements illegal in the future. > >> > > >> > Ok, but I still don't understand why the cdefs were removed from the > >> > proposed structure members, when they are required for class > >> > definitions. > >> > >> Because structs can only have c members, so the cdef was entirely > >> redundant. They weren't really removed per say, it's just that with > >> the exception of cdef classes, "cdef ..." meant "a c declaration > >> follows." > > > > But cdef classes can also only have cdef members. > > > > I think it's better to keep cdef meaning "backed by C data", not > > necessarily "written using C syntax", since you're trying to do more > > with Cython, so it doesn't make sense to force C syntax. > > It means both. Were I to start over, I would make "cdef int* a, b, c" > declare three pointers, but we're stuck with the C syntax we have... > > In a struct "cdef" on members is entirely redundant As it is in a cdef class. If you don't want it there for struct/union members, though, then I'll do it that way. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sun Feb 20 06:21:31 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 19 Feb 2011 21:21:31 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110220023207.GA25084@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110220023207.GA25084@tyr.home.net> Message-ID: On Sat, Feb 19, 2011 at 6:32 PM, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote: >> > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >> >> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >> >> > On Sat, Feb 19, 2011 at 12:47:41PM -0800, Robert Bradshaw wrote: >> >> >> On Sat, Feb 19, 2011 at 11:22 AM, W. Trevor King wrote: >> >> >> > However, the filename <-> module mapping is troublesome for backing >> >> >> > externally-implemented Python modules (e.g. numpy). ?If you wanted to >> >> >> > write a .pxd file backing numpy.random, how would you go about getting >> >> >> > your module installed in Cython/Includes/numpy/random.pxd or another >> >> >> > path that cython would successfully match with `cimport numpy.random`? >> >> >> >> >> >> Note that extern blocks (by definition) declare where things come from. >> >> > >> >> > They declare where the .pxd file looks for .h files, but not where >> >> > .pyx files look for the .pxd file. >> >> >> >> Sorry, I should have said extern blocks that make cdef class >> >> declarations (such as our numpy.pxd). >> > >> > It doesn't look like there are cdef class declarations in numpy.pxd: >> > >> > ? ?cython $ grep class Cython/Includes/numpy.pxd >> > ? ? ? ?ctypedef class numpy.dtype [object PyArray_Descr]: >> > ? ? ? ?ctypedef extern class numpy.flatiter [object PyArrayIterObject]: >> > ? ? ? ?ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: >> > ? ? ? ?ctypedef class numpy.ndarray [object PyArrayObject]: >> > ? ? ? ?ctypedef extern class numpy.ufunc [object PyUFuncObject]: >> > >> > This still doesn't explain how .pxd files specify which external >> > implemented Python modules they correspond to. >> >> "numpy.dtype" is the fully qualified name of the class it's >> declaring--the module is "numpy." > > Hmm, that means that > > ? ?cimport numpy > ? ?a = numpy.numpy.ndarray > > also compiles. As does a = numpy.foo.x.y.z though perhaps that should be a compile-time error. To get the type, you have to write cimport numpy a = numpy.ndarray. > Wouldn't it make more sense to mirror numpy's module > structure when backing it? ?You do have nested .pxd modules for > cpython, libc, etc. Yes, in fact that's what we do, but that's not required (e.g. you can have these extern declarations in .pyx files) >> >> >> >> > ? ? ?cdef public struct X: >> >> >> >> > ? ? ? ? ?int x >> >> >> >> > ? ? ? ? ?readonly int z >> >> >> >> > ? ? ? ? ?private int z >> >> >> >> > >> >> >> >> > I would perhaps say that non-Pythonable non-private members in public >> >> >> >> > structs would be a compile error. >> >> >> >> >> >> >> >> +1, keep it safe at the beginning. >> >> >> > >> >> >> > -1, keep the code clean and the interface consistent ;). ?I think the >> >> >> > struct syntax should be identical to the class syntax, with the >> >> >> > exception that you can't bind methods to structs. ?That's the only >> >> >> > real difference between structs and classes, isn't it? >> >> >> >> >> >> In C++, the only difference between structs and classes is that struct >> >> >> members are public by default. (Not saying that C++ is always the >> >> >> model to follow, but it gives precedent). And structs can have >> >> >> function members, that's how to do OOP in C. >> >> > >> >> > Oh. ?Even more reason to have identical struct and class handling in >> >> > Cython ;). >> >> > >> >> > It is unclear to me what `cdef public struct` means. ?I think it >> >> > should mean "Python bindings can alter this struct's definition", >> >> > which doesn't make sense. >> >> >> >> I think it should mean "this struct is accessible from Python (as X)" >> > >> > Wouldn't that be "cdef readonly struct X"? >> > >> >> > Shouldn't the syntax for public members be >> >> > >> >> > ? ?cdef struct X: >> >> > ? ? ? ?cdef public: >> >> > ? ? ? ? ? ?int x >> >> > ? ? ? ? ? ?readonly int y >> >> > ? ? ? ? ? ?private int z >> >> >> >> -1 on nesting things like this. Rather than make a struct visible from >> >> Python iff any of its members are, >> > >> > A struct is visible from python iff it is declared public or readonly: >> > >> > ? ?cdef public struct X: >> > ? ? ? ?... >> > >> > or >> > >> > ? ?cdef readonly struct X: >> > ? ? ? ?... >> > >> > I don't think the visibility of the struct as a whole should have any >> > effect over the visibility of its members, so you should be able to >> > specify member visibility explicitly with per-member granularity (as >> > you currently can for classes). >> > >> > I was assuming that structs would be public by default (like classes), >> > but that is obviously configurable. >> > >> >> I think it makes more sense to put >> >> the declaration on the struct itself. We could support >> >> >> >> cdef public struct X: >> >> ? ? int x # public >> >> >> >> cdef readonly struct Y: >> >> ? ? int y # readonly >> >> >> >> cdef [private] struct Z: >> >> ? ? int z # private, as we don't even have Z in the Python namespace, >> >> and no wrapper is created. >> > >> > The problems with this are: >> > >> > * It's differnent from how we handle the almost identical class case. >> >> True it's different, but there are significant differences between >> classes and structs in Cython, and this is the way things are now. > > Ah, well. ?I suppose I'll leave the Parser pretty much alone. ?Classes > and structs will end up being much closer on the Python interface > side, perhaps the syntax will grow closer in future Cythons... Well, we certainly can't get rid of the old syntax without a lot of thought and justification. >> > * It makes it impossible to define, for example a public struct with >> > ?C-only attributes: >> > >> > ? ?cdef public struct X: >> > ? ? ? ?cdef public int a >> > ? ? ? ?cdef private void* ptr >> >> ? >> >> The above would work just fine. I was proposing that it would be >> semantically equivalent to >> >> ? ? cdef public struct X: >> ? ? ? ? cdef int a >> ? ? ? ? cdef private void* ptr >> >> Perhaps that was not clear. Consider the currently valid >> >> cdef struct X: >> ? ? int a >> ? ? void* ptr >> >> We can't make everything public by default, as this would break valid >> code. (I don't think it's a good idea to make the default visibility >> of a member be a function of its type if we can help it.) > > Oh, I though (for some reason) that you were ruling out member-level > visibility adjustments. ?Using the struct visibility to set > member-level defaults is fine. > >> BTW, the "public" keyword is the wrong thing to use here, as that >> actually controls name mangling and (c-level) symbol exporting. The >> fact that means a different thing for members than for top-level >> symbols isn't ideal, but at least it's unambiguous as members need not >> be mangled. > > "public" means "read/write Python interface" for struct attributes, "public" means "C-visible" for global names, "read/write Python interface" for (cdef class) members. > I > was assuming that would also apply to struct members acording to > tutorial/cdef_classes. ?Should I use a different visibility name in > structs and unions? No, I think public/readonliy/private is a perfectly fine thing for struct members. It's just that declaring the struct itself as public already has a different meaning. >> >> >> > If safety with a new feature is a concern, a warning like >> >> >> > "EXPERIMENTAL FEATURE" in the associated docs and compiler output >> >> >> > should be sufficient. >> >> >> >> >> >> I think the point of "safe" is to start out with a compiler error, and >> >> >> we can change our minds later, which is better than trying to make >> >> >> legal statements illegal in the future. >> >> > >> >> > Ok, but I still don't understand why the cdefs were removed from the >> >> > proposed structure members, when they are required for class >> >> > definitions. >> >> >> >> Because structs can only have c members, so the cdef was entirely >> >> redundant. They weren't really removed per say, it's just that with >> >> the exception of cdef classes, "cdef ..." meant "a c declaration >> >> follows." >> > >> > But cdef classes can also only have cdef members. >> > >> > I think it's better to keep cdef meaning "backed by C data", not >> > necessarily "written using C syntax", since you're trying to do more >> > with Cython, so it doesn't make sense to force C syntax. >> >> It means both. Were I to start over, I would make "cdef int* a, b, c" >> declare three pointers, but we're stuck with the C syntax we have... >> >> In a struct "cdef" on members is entirely redundant > > As it is in a cdef class. ?If you don't want it there for struct/union > members, though, then I'll do it that way. We're sticking with backwards compatibility. - Robert From stefan_ml at behnel.de Sun Feb 20 09:40:56 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Feb 2011 09:40:56 +0100 Subject: [Cython] Gmane archive In-Reply-To: <4D5DFA97.4010709@behnel.de> References: <4D5DFA97.4010709@behnel.de> Message-ID: <4D60D398.9080903@behnel.de> Stefan Behnel, 18.02.2011 05:50: > I still didn't get a response from Gmane, but this article doesn't look > promising: > > http://article.gmane.org/gmane.discuss/13987 > > So I guess we'll have to request a new group. Very unfortunate. They've updated the group subscription. Turns out that changing Gmane group *names* is a problem, not their subscribed e-mail address. And the best place to ask for updates is not via e-mail (as the FAQ suggests), but via the gmane.discuss group. Stefan From stefan_ml at behnel.de Sun Feb 20 10:26:20 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Feb 2011 10:26:20 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110219233126.GC25132@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> Message-ID: <4D60DE3C.3070306@behnel.de> W. Trevor King, 20.02.2011 00:31: > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >>> It is unclear to me what `cdef public struct` means. I think it >>> should mean "Python bindings can alter this struct's definition", >>> which doesn't make sense. >> >> I think it should mean "this struct is accessible from Python (as X)" > > Wouldn't that be "cdef readonly struct X"? The problem here is that "public" is used differently in different contexts - usually "exported at the C level" in this kind of context, with the quirk of meaning "modifiable at the Python level" for cdef class attributes. The potentially clearer "cpdef" means "overridable at the Python level" as well as "visible at the Python level", so it doesn't quite match the second meaning by itself. Structs won't be overridable at the Python level, I guess, so cpdef isn't quite right. The intention here isn't to export them at the C level either, so "public" isn't right. We could tweak the "cpdef" meaning into "cdef thing mapped to the Python level, and overridable if supported in the given context", which would lead to broader applicability. Then we could allow cpdef readonly struct ... I think that's a lot clearer than adding to the double meaning of "public". I would also prefer if Python accessible cdef class attributes were defined using "cpdef". We could potentially make that the preferred way in the future? Stefan From stefan_ml at behnel.de Sun Feb 20 10:38:19 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Feb 2011 10:38:19 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110220023207.GA25084@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110220023207.GA25084@tyr.home.net> Message-ID: <4D60E10B.9070106@behnel.de> W. Trevor King, 20.02.2011 03:32: > On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote: >> On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote: >>> I think it's better to keep cdef meaning "backed by C data", not >>> necessarily "written using C syntax", since you're trying to do more >>> with Cython, so it doesn't make sense to force C syntax. >> >> It means both. Were I to start over, I would make "cdef int* a, b, c" >> declare three pointers +100 > but we're stuck with the C syntax we have... Sadly, yes. This will be impossible to change in the future without breaking all sorts of code in a hard-to-fix-automatically way. >> In a struct "cdef" on members is entirely redundant > > As it is in a cdef class. Not quite. The class body of (cdef) classes can potentially contain code, so it's important for the parser to know what is a declaration and what is an executable statement. Stefan From vitja.makarov at gmail.com Sun Feb 20 18:23:10 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 20 Feb 2011 20:23:10 +0300 Subject: [Cython] Control flow graph In-Reply-To: References: <4D5A2D6E.9060406@behnel.de> Message-ID: 2011/2/16 Vitja Makarov : > 2011/2/15 Stefan Behnel : >> Robert Bradshaw, 15.02.2011 08:21: >>> >>> On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote: >>>> >>>> 2011/2/15 Robert Bradshaw: >>>>> >>>>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote: >>>>>> >>>>>> Hi! >>>>>> >>>>>> In order to implement "reaching definitions" algorithm. >>>>>> I'm now working on control-flow (or data-flow) graph. >>>>>> >>>>>> Here is funny picture made with graphviz ;) >>>>>> >>>>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/ >>>>> >>>>> Cool. Any plans on handling exceptions? >>>> >>>> Sure, but I don't have much time for this :( >>>> >>>> Linear block inside try...except body should be split by assignments >>>> and each subblock should point to exception handling entry point. >>> >>> Would every possible failing sub-expression have to point to the >>> exception handling point(s)? >> >> Well, in most cases (especially the interesting ones), this will be the >> function exit point, so it'll be easy. And in some cases, we may be able to >> infer that a specific exception that an expression (e.g. arithmetics or a >> 'raise' statement) can raise will not get caught by a given except clause >> (although that's certainly a tricky optimisation). >> >> But in general, I think any subexpression that potentially raises an >> exception must point to the next exception handling point. >> >> >>> I suppose it depends on whether you'll be handling more than assignment >>> tracking. >> >> We *may* get away with a statement-level graph in that case, but I somehow >> doubt it already. For example, list comprehensions leak their variable in >> Py2 code, so it's important to know if they are executed or not, and they >> may appear in any kind of expression. >> > > Hmm... both python and codespeaks in the thread > > Here is my commit it's mostly broken now but anyway > https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a > > I've update stuff: - algo for finding definitions - warnings for uninitialized and may be uninitialised use - few test cases Trying to compile ParseTreeTransforms.py I've found this for example: warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable 'template' may be used uninitialized def create_Property(self, entry): if entry.visibility == 'public': if entry.type.is_pyobject: template = self.basic_pyobject_property else: template = self.basic_property elif entry.visibility == 'readonly': template = self.basic_property_ro property = template.substitute({ u"ATTR": ExprNodes.AttributeNode(pos=entry.pos, obj=ExprNodes.NameNode(pos=entry.pos, name="self"), attribute=entry.name), }, pos=entry.pos).stats[0] -- vitja. From greg.ewing at canterbury.ac.nz Sun Feb 20 21:13:49 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 21 Feb 2011 09:13:49 +1300 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> Message-ID: <4D6175FD.30700@canterbury.ac.nz> Robert Bradshaw wrote: > BTW, the "public" keyword is the wrong thing to use here, as that > actually controls name mangling and (c-level) symbol exporting. The > fact that means a different thing for members than for top-level > symbols isn't ideal, but at least it's unambiguous as members need not > be mangled. The overloading of 'public' is really a bit of a mess. I've been thinking for a while that there really ought to be a different keyword such as "exposed" for declaring that things are to be exposed to Python. It would be useful in lots of ways: cdef class Foo: cdef exposed int i # formerly 'public' cdef exposed enum E: a, b, c # creates Python bindings for these names cdef exposed struct S: # Exposed but mangled as usual ... cdef public exposed struct S: # Exposed and unmangled ... -- Greg From greg.ewing at canterbury.ac.nz Sun Feb 20 22:09:42 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 21 Feb 2011 10:09:42 +1300 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110220023207.GA25084@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110220023207.GA25084@tyr.home.net> Message-ID: <4D618316.1050707@canterbury.ac.nz> W. Trevor King wrote: > On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote: > >>On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King wrote: >> >>> cython $ grep class Cython/Includes/numpy.pxd >>> ctypedef class numpy.dtype [object PyArray_Descr]: >>> ctypedef extern class numpy.flatiter [object PyArrayIterObject]: >>> ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]: >>> ctypedef class numpy.ndarray [object PyArrayObject]: >>> ctypedef extern class numpy.ufunc [object PyUFuncObject]: >> >>"numpy.dtype" is the fully qualified name of the class it's >>declaring--the module is "numpy." Aren't the module names in the class declarations redundant here? Since they're in a file called numpy.pxd, declarations will go into the numpy module namespace by default. You should be able to write these as just ctypedef class dtype [object PyArray_Descr]: ctypedef class flatiter [object PyArrayIterObject]: ctypedef class broadcast [object PyArrayMultiIterObject]: ctypedef class ndarray [object PyArrayObject]: ctypedef class ufunc [object PyUFuncObject]: The 'extern' syntax for extension classes, including a module name, is really an obsolete feature. Before Pyrex had .pxd files, it was the only way to declare an externally implemented extension type to Pyrex. But now that .pxd files exist, there should be no need for it. > Hmm, that means that > > cimport numpy > a = numpy.numpy.ndarray > > also compiles. Compiles and runs, or just compiles? If this works as a way of getting hold of the ndarray type, then it's a bug -- it's not supposed to work that way. -- Greg From stefan_ml at behnel.de Sun Feb 20 22:17:58 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 20 Feb 2011 22:17:58 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D6175FD.30700@canterbury.ac.nz> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D6175FD.30700@canterbury.ac.nz> Message-ID: <4D618506.6070702@behnel.de> Greg Ewing, 20.02.2011 21:13: > Robert Bradshaw wrote: > >> BTW, the "public" keyword is the wrong thing to use here, as that >> actually controls name mangling and (c-level) symbol exporting. The >> fact that means a different thing for members than for top-level >> symbols isn't ideal, but at least it's unambiguous as members need not >> be mangled. > > The overloading of 'public' is really a bit of a mess. I've been > thinking for a while that there really ought to be a different > keyword such as "exposed" for declaring that things are to be > exposed to Python. It would be useful in lots of ways: > > cdef class Foo: > cdef exposed int i # formerly 'public' > > cdef exposed enum E: > a, b, c # creates Python bindings for these names > > cdef exposed struct S: # Exposed but mangled as usual > ... > > cdef public exposed struct S: # Exposed and unmangled > ... Given that Cython has "cpdef" already, why not just use that? Stefan From wking at drexel.edu Mon Feb 21 01:09:37 2011 From: wking at drexel.edu (W. Trevor King) Date: Sun, 20 Feb 2011 19:09:37 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D618316.1050707@canterbury.ac.nz> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110220023207.GA25084@tyr.home.net> <4D618316.1050707@canterbury.ac.nz> Message-ID: <20110221000937.GA29663@tyr.home.net> On Mon, Feb 21, 2011 at 10:09:42AM +1300, Greg Ewing wrote: > W. Trevor King wrote: > > Hmm, that means that > > > > cimport numpy > > a = numpy.numpy.ndarray > > > > also compiles. > > Compiles and runs, or just compiles? You're right. Cython compilation worked, but it didn't run: $ export CFLAGS="-I/usr/lib/python2.6/site-packages/numpy/core/include/" $ python -c 'import pyximport; pyximport.install(); import np;' Traceback (most recent call last): ... File "np.pyx", line 3, in init np (/.../np.c:2846) a = numpy.numpy.ndarray ImportError: Building module failed: ['NameError: numpy\n'] Sorry for muddying the waters with that claim. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From greg.ewing at canterbury.ac.nz Mon Feb 21 11:40:10 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 21 Feb 2011 23:40:10 +1300 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D618506.6070702@behnel.de> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D6175FD.30700@canterbury.ac.nz> <4D618506.6070702@behnel.de> Message-ID: <4D62410A.3030808@canterbury.ac.nz> Stefan Behnel wrote: > Given that Cython has "cpdef" already, why not just use that? That seems like a reasonable idea. -- Greg From robertwb at math.washington.edu Mon Feb 21 19:11:59 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 21 Feb 2011 10:11:59 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D60DE3C.3070306@behnel.de> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D60DE3C.3070306@behnel.de> Message-ID: On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote: > W. Trevor King, 20.02.2011 00:31: >> >> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >>> >>> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >>>> >>>> It is unclear to me what `cdef public struct` means. ?I think it >>>> should mean "Python bindings can alter this struct's definition", >>>> which doesn't make sense. >>> >>> I think it should mean "this struct is accessible from Python (as X)" >> >> Wouldn't that be "cdef readonly struct X"? > > The problem here is that "public" is used differently in different contexts > - usually "exported at the C level" in this kind of context, with the quirk > of meaning "modifiable at the Python level" for cdef class attributes. > > The potentially clearer "cpdef" means "overridable at the Python level" as > well as "visible at the Python level", so it doesn't quite match the second > meaning by itself. > > Structs won't be overridable at the Python level, I guess, so cpdef isn't > quite right. The intention here isn't to export them at the C level either, > so "public" isn't right. > > We could tweak the "cpdef" meaning into "cdef thing mapped to the Python > level, and overridable if supported in the given context", which would lead > to broader applicability. Then we could allow That's what I was thinking. > ? ?cpdef readonly struct ... > > I think that's a lot clearer than adding to the double meaning of "public". > I would also prefer if Python accessible cdef class attributes were defined > using "cpdef". We could potentially make that the preferred way in the > future? We could allow it, but -1 to disallowing "cdef class" - Robert From stefan_ml at behnel.de Mon Feb 21 19:26:52 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 21 Feb 2011 19:26:52 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D60DE3C.3070306@behnel.de> Message-ID: <4D62AE6C.5040105@behnel.de> Robert Bradshaw, 21.02.2011 19:11: > On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote: >> W. Trevor King, 20.02.2011 00:31: >>> >>> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >>>> >>>> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >>>>> >>>>> It is unclear to me what `cdef public struct` means. I think it >>>>> should mean "Python bindings can alter this struct's definition", >>>>> which doesn't make sense. >>>> >>>> I think it should mean "this struct is accessible from Python (as X)" >>> >>> Wouldn't that be "cdef readonly struct X"? >> >> The problem here is that "public" is used differently in different contexts >> - usually "exported at the C level" in this kind of context, with the quirk >> of meaning "modifiable at the Python level" for cdef class attributes. >> >> The potentially clearer "cpdef" means "overridable at the Python level" as >> well as "visible at the Python level", so it doesn't quite match the second >> meaning by itself. >> >> Structs won't be overridable at the Python level, I guess, so cpdef isn't >> quite right. The intention here isn't to export them at the C level either, >> so "public" isn't right. >> >> We could tweak the "cpdef" meaning into "cdef thing mapped to the Python >> level, and overridable if supported in the given context", which would lead >> to broader applicability. Then we could allow > > That's what I was thinking. > >> cpdef readonly struct ... >> >> I think that's a lot clearer than adding to the double meaning of "public". >> I would also prefer if Python accessible cdef class attributes were defined >> using "cpdef". We could potentially make that the preferred way in the >> future? > > We could allow it, but -1 to disallowing "cdef class" With "preferred way", I was suggesting that we could *deprecate* cdef public int x cdef readonly object y for cdef class properties in favour of cpdef int x cpdef readonly object y and change the documentation accordingly etc., so that at least new users get used to the new way. The old way would likely continue to be supported until Cython 2.0 or so, for the holy cow's sake... Stefan From dalcinl at gmail.com Mon Feb 21 19:38:06 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 21 Feb 2011 15:38:06 -0300 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D62AE6C.5040105@behnel.de> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D60DE3C.3070306@behnel.de> <4D62AE6C.5040105@behnel.de> Message-ID: On 21 February 2011 15:26, Stefan Behnel wrote: > Robert Bradshaw, 21.02.2011 19:11: >> >> On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote: >>> >>> W. Trevor King, 20.02.2011 00:31: >>>> >>>> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >>>>> >>>>> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >>>>>> >>>>>> It is unclear to me what `cdef public struct` means. ?I think it >>>>>> should mean "Python bindings can alter this struct's definition", >>>>>> which doesn't make sense. >>>>> >>>>> I think it should mean "this struct is accessible from Python (as X)" >>>> >>>> Wouldn't that be "cdef readonly struct X"? >>> >>> The problem here is that "public" is used differently in different >>> contexts >>> - usually "exported at the C level" in this kind of context, with the >>> quirk >>> of meaning "modifiable at the Python level" for cdef class attributes. >>> >>> The potentially clearer "cpdef" means "overridable at the Python level" >>> as >>> well as "visible at the Python level", so it doesn't quite match the >>> second >>> meaning by itself. >>> >>> Structs won't be overridable at the Python level, I guess, so cpdef isn't >>> quite right. The intention here isn't to export them at the C level >>> either, >>> so "public" isn't right. >>> >>> We could tweak the "cpdef" meaning into "cdef thing mapped to the Python >>> level, and overridable if supported in the given context", which would >>> lead >>> to broader applicability. Then we could allow >> >> That's what I was thinking. >> >>> ? ?cpdef readonly struct ... >>> >>> I think that's a lot clearer than adding to the double meaning of >>> "public". >>> I would also prefer if Python accessible cdef class attributes were >>> defined >>> using "cpdef". We could potentially make that the preferred way in the >>> future? >> >> We could allow it, but -1 to disallowing "cdef class" > > With "preferred way", I was suggesting that we could *deprecate* > > ? ?cdef public int x > ? ?cdef readonly object y > > for cdef class properties in favour of > > ? ?cpdef int x > ? ?cpdef readonly object y > > and change the documentation accordingly etc., so that at least new users > get used to the new way. +1 > The old way would likely continue to be supported > until Cython 2.0 or so, for the holy cow's sake... > But with a deprecation warning, right? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Mon Feb 21 19:38:47 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 21 Feb 2011 10:38:47 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D62AE6C.5040105@behnel.de> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D60DE3C.3070306@behnel.de> <4D62AE6C.5040105@behnel.de> Message-ID: On Mon, Feb 21, 2011 at 10:26 AM, Stefan Behnel wrote: > Robert Bradshaw, 21.02.2011 19:11: >> >> On Sun, Feb 20, 2011 at 1:26 AM, Stefan Behnel wrote: >>> >>> W. Trevor King, 20.02.2011 00:31: >>>> >>>> On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: >>>>> >>>>> On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: >>>>>> >>>>>> It is unclear to me what `cdef public struct` means. ?I think it >>>>>> should mean "Python bindings can alter this struct's definition", >>>>>> which doesn't make sense. >>>>> >>>>> I think it should mean "this struct is accessible from Python (as X)" >>>> >>>> Wouldn't that be "cdef readonly struct X"? >>> >>> The problem here is that "public" is used differently in different >>> contexts >>> - usually "exported at the C level" in this kind of context, with the >>> quirk >>> of meaning "modifiable at the Python level" for cdef class attributes. >>> >>> The potentially clearer "cpdef" means "overridable at the Python level" >>> as >>> well as "visible at the Python level", so it doesn't quite match the >>> second >>> meaning by itself. >>> >>> Structs won't be overridable at the Python level, I guess, so cpdef isn't >>> quite right. The intention here isn't to export them at the C level >>> either, >>> so "public" isn't right. >>> >>> We could tweak the "cpdef" meaning into "cdef thing mapped to the Python >>> level, and overridable if supported in the given context", which would >>> lead >>> to broader applicability. Then we could allow >> >> That's what I was thinking. >> >>> ? ?cpdef readonly struct ... >>> >>> I think that's a lot clearer than adding to the double meaning of >>> "public". >>> I would also prefer if Python accessible cdef class attributes were >>> defined >>> using "cpdef". We could potentially make that the preferred way in the >>> future? >> >> We could allow it, but -1 to disallowing "cdef class" > > With "preferred way", I was suggesting that we could *deprecate* > > ? ?cdef public int x > ? ?cdef readonly object y > > for cdef class properties in favour of > > ? ?cpdef int x > ? ?cpdef readonly object y Oh, you were talking about members. For sure. > and change the documentation accordingly etc., so that at least new users > get used to the new way. The old way would likely continue to be supported > until Cython 2.0 or so, for the holy cow's sake... :) This kind of thing is much easier to fix than, say, fallout from http://trac.cython.org/cython_trac/ticket/654 - Robert From greg.ewing at canterbury.ac.nz Mon Feb 21 22:12:52 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 22 Feb 2011 10:12:52 +1300 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D62AE6C.5040105@behnel.de> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D60DE3C.3070306@behnel.de> <4D62AE6C.5040105@behnel.de> Message-ID: <4D62D554.1050109@canterbury.ac.nz> Stefan Behnel wrote: > With "preferred way", I was suggesting that we could *deprecate* > > cdef public int x > cdef readonly object y > > for cdef class properties in favour of > > cpdef int x > cpdef readonly object y I think I've just realised one of the reasons for my gut dislike of the "cpdef" keyword -- it looks too similar to "def". At first glance, it's hard to spot the difference between the cdef and cpdef versions of two otherwise identical declarations. I think this is too subtle for something that makes such a big difference to semantics. It's often important that Python code is not allowed to mess with an object's internal state, so making it possible should require something more obvious. -- Greg From stefan_ml at behnel.de Mon Feb 21 22:22:24 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 21 Feb 2011 22:22:24 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D62D554.1050109@canterbury.ac.nz> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D60DE3C.3070306@behnel.de> <4D62AE6C.5040105@behnel.de> <4D62D554.1050109@canterbury.ac.nz> Message-ID: <4D62D790.8020907@behnel.de> Greg Ewing, 21.02.2011 22:12: > Stefan Behnel wrote: > >> With "preferred way", I was suggesting that we could *deprecate* >> >> cdef public int x >> cdef readonly object y >> >> for cdef class properties in favour of >> >> cpdef int x >> cpdef readonly object y > > I think I've just realised one of the reasons for my gut > dislike of the "cpdef" keyword -- it looks too similar to > "def". At first glance, it's hard to spot the difference > between the cdef and cpdef versions of two otherwise > identical declarations. > > I think this is too subtle for something that makes such > a big difference to semantics. It's often important that > Python code is not allowed to mess with an object's > internal state, so making it possible should require > something more obvious. Hmm, I don't know. Maybe I'm just used to it already, but I don't find it hard to spot at all. The same argument could be brought up against "cdef" vs. "def" (between which the semantic difference is *huge*), and I hope you don't find that hard to spot. Stefan From greg.ewing at canterbury.ac.nz Mon Feb 21 22:46:22 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 22 Feb 2011 10:46:22 +1300 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D62D790.8020907@behnel.de> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <4D60DE3C.3070306@behnel.de> <4D62AE6C.5040105@behnel.de> <4D62D554.1050109@canterbury.ac.nz> <4D62D790.8020907@behnel.de> Message-ID: <4D62DD2E.3000906@canterbury.ac.nz> Stefan Behnel wrote: > The same argument could be brought up against > "cdef" vs. "def" (between which the semantic difference is *huge*) There are a couple of differences: - 'cdef' and 'def' look very different (at least to me) because they *start* with a different letter. Whereas 'cdef' and 'cpdef' both start and end with the same letter, maknig tehm mcuh eazier to conufse. - There is much less semantic overlap betwen 'def' and 'cdef'. If you use the wrong one, you usually find out about it fairly quickly one way or another -- you get a syntax error (e.g. 'def struct'), or something doesn't work the way you expected (e.g. a function you thought you had exposed doesn't show up in Python). It's possible to accidentally expose a function, but only if you declare it with an implicit return type, which I never do when writing what I intend to be a C function (this could perhaps be made illegal to further reduce the chances of such an error). Mistaken use of 'cpdef' on a C attribute, on the other hand, would very often pass silently. -- Greg From wking at drexel.edu Tue Feb 22 18:55:43 2011 From: wking at drexel.edu (W. Trevor King) Date: Tue, 22 Feb 2011 12:55:43 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110219233126.GC25132@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> Message-ID: <20110222175542.GA8312@tyr.home.net> On Sat, Feb 19, 2011 at 06:31:26PM -0500, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: > > On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: > > > Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to > > > consolidate the way these are handled in Parsing/Nodes/Symtab and I > > > want to make sure I don't implement the wrong interpretation. Can you > > > clarify how one knows if "public" means "expose a read/write Python > > > interface to this object" or "expose this symbol to external C code"? > > > > Public has had several different meanings. I wish there were a spec > > and full grammer for Cython, but there's not (yet?). The meaning is > > implicit in the code, and there's enough users out there that we > > should stay backwards compatible. It may be worth doing some > > backwards-incompatible normalization before we hit 1.0, but compiling > > the entire Python grammar is higher priority than that. > > Since I'm going to have lots of similar stuff (classes, enums, > structs, unions) all with the same (hopefully) cdef/cpdef/visibility > stuff for members, I'd like to consolidate now. I will of course, add > special-case code as necessary to support the current syntax, which > can then be removed whenever you think it is appropriate, but writing > separate, near-identical handlers for each type seems like a recipe > for disaster ;). > > I'll look to the code for guidance on public, and try to work out the > appropriate meaning during the parse phase. I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take: git: http://www.physics.drexel.edu/~wking/code/git/cython.git branch: cdef-enums-stucts-and-unions gitweb: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;h=refs/heads/cdef-enums-stucts-and-unions -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From stefan_ml at behnel.de Tue Feb 22 20:18:21 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Feb 2011 20:18:21 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110222175542.GA8312@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> Message-ID: <4D640BFD.4050708@behnel.de> W. Trevor King, 22.02.2011 18:55: > I've been working on a more explicit parser that removes the ambiguity > behind the various visibilities. This will help me ensure proper > impolementation of my cdef-ed enums/structs/..., and make it easier to > update visibility syntax in the future. Take a look and let me know > if you think this is a useful direction to take: > > git: http://www.physics.drexel.edu/~wking/code/git/cython.git > branch: cdef-enums-stucts-and-unions > gitweb: > http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;h=refs/heads/cdef-enums-stucts-and-unions It doesn't seem like I can leave comments in the gitweb version, so I'll comment here. First thing that caught my eyes: I hope you do not intend to leave the logging usage in the parser. This is seriously performance critical code that Cython compiles down to pretty fast C code. Note that you can use a debugger and Python's profiling/tracing interface to find out what's happening, without code impact. Some of the log statements span more than one line, which makes it trickier to strip them out with sed&friends (but backing out the initial changeset would likely make it simple enough to remove the rest manually). Also note that it's best to write runnable tests ("tests/run/"). The tests in "tests/compile/" are only compiled and imported. See the hacking guide in the wiki. I know you're not there yet with your implementation, I'm just mentioning it. Most important point, however: I think it's a good idea to clean up the parsing context the way you did it. The semantic distinction of the three new classes you added makes sense to me. CtxAttribute is the wrong name, though. And the class copy implementation gets even more generic than it already was in the Ctx. I'm not a big fan of that, especially not in the parser. For one, it prevents changing the classes into cdef classes, which had long been on my list for Ctx. CSource: doesn't sound like quite the right name - it does not describe a C source file but information that Cython has about non-Cython things. I also doubt that Cython allows you to call an attribute "cdef", you'll need to change that. Stefan From wking at drexel.edu Tue Feb 22 21:02:08 2011 From: wking at drexel.edu (W. Trevor King) Date: Tue, 22 Feb 2011 15:02:08 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D640BFD.4050708@behnel.de> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> Message-ID: <20110222200207.GA14712@tyr.home.net> On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote: > W. Trevor King, 22.02.2011 18:55: > > I've been working on a more explicit parser that removes the ambiguity > > behind the various visibilities. This will help me ensure proper > > impolementation of my cdef-ed enums/structs/..., and make it easier to > > update visibility syntax in the future. Take a look and let me know > > if you think this is a useful direction to take: > > First thing that caught my eyes: I hope you do not intend to leave the > logging usage in the parser. This is seriously performance critical code > that Cython compiles down to pretty fast C code. Note that you can use a > debugger and Python's profiling/tracing interface to find out what's > happening, without code impact. I can strip them out afterwards, but it helps me figure out what I've broken if I shift too much around at the same time. I don't know enough about Python's trace module to know if I can turn on tracing only for functions defined in a single module or not, since otherwise its hard for me to separate signal from noise. > Some of the log statements span more than one line, which makes it trickier > to strip them out with sed&friends (but backing out the initial changeset > would likely make it simple enough to remove the rest manually). Hmm, perhaps I'll condense the logging statements down onto one (long) line a piece, that will make it easy to comment/uncomment them with sed/emacs/etc. I suppose once Cython can compile the logging module we could leave them in with reduced overhead ;). > Also note that it's best to write runnable tests ("tests/run/"). The > tests in "tests/compile/" are only compiled and imported. See the > hacking guide in the wiki. I know you're not there yet with your > implementation, I'm just mentioning it. Thanks for the tip. > CtxAttribute is the wrong name, though. And the class copy > implementation gets even more generic than it already was in the > Ctx. I'm not a big fan of that, especially not in the parser. For > one, it prevents changing the classes into cdef classes, which had > long been on my list for Ctx. An easy, if uglier, workaround would be to prepend attributes with the class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. Then the Ctx class could subclass the current CtxAttribute classes instead of binding instances of each of them. That way Ctx would keep its traditional flat attribute namespace and easy deepcopy, but eveyone in Nodes, etc. that will use the attributes would become class-name dependent. The CtxAttribute class is, as its docstring says, just a hook for its deepcopy method. With an alternative deepcopy implementation, CtxAttribute could be replaced with the standard `object`, so don't worry too much about its name at this point ;). > CSource: doesn't sound like quite the right name - it does not describe a C > source file but information that Cython has about non-Cython things. It's a container for attributes that describe the presence and location of backing C definitions. * cdef: "Will there be a backing C defintion? * extern: "Has someone else already written it?" * name/namespace: "What did they call it?" If you'd rather I called the class something else, I'm certainly willing to change it. > I also doubt that Cython allows you to call an attribute "cdef", you'll > need to change that. It seems to work for me: >>> import Cython.Compiler.Parsing as P >>> P.__file__ 'Cython/Compiler/Parsing.so' >>> c = P.CSource() >>> dir(c) [..., 'cdef', 'deepcopy', 'extern', 'name', 'namespace'] >>> c.cdef 0 >>> c.cdef = 1 >>> c.cdef 1 However, I agree that it's generally a bad idea to play around with keywords. I'll revert it to the wordier-but-less-confusing `cdef_flag`. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From stefan_ml at behnel.de Tue Feb 22 21:27:18 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Feb 2011 21:27:18 +0100 Subject: [Cython] Control flow graph In-Reply-To: References: <4D5A2D6E.9060406@behnel.de> Message-ID: <4D641C26.9070705@behnel.de> Vitja Makarov, 20.02.2011 18:23: > 2011/2/16 Vitja Makarov: >> Hmm... both python and codespeaks in the thread Yes, we should keep it to cython-devel only. Sorry for mixing it up. >> Here is my commit it's mostly broken now but anyway >> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a Flow control support is large enough to merit its own module. Not sure how 'smart' git is here, but you can always keep the history by explicitly copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated sections from both files. You are duplicating some code from the type inferencer. We might want to clean that up at some point. However, given that flow control analysis will allow us to improve the type inferencer, I think it's best to keep this code in the FCA part. > I've update stuff: > - algo for finding definitions > - warnings for uninitialized and may be uninitialised use > - few test cases That looks very nice so far. Any idea how well it scales? > Trying to compile ParseTreeTransforms.py I've found this for example: > > warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable > 'template' may be used uninitialized > > def create_Property(self, entry): > if entry.visibility == 'public': > if entry.type.is_pyobject: > template = self.basic_pyobject_property > else: > template = self.basic_property > elif entry.visibility == 'readonly': > template = self.basic_property_ro > property = template.substitute({ > u"ATTR": ExprNodes.AttributeNode(pos=entry.pos, > > obj=ExprNodes.NameNode(pos=entry.pos, name="self"), > attribute=entry.name), > }, pos=entry.pos).stats[0] Ok, I guess that code generally works, but it's better to get rid of the code smell. Stefan From dalcinl at gmail.com Tue Feb 22 21:41:22 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 22 Feb 2011 17:41:22 -0300 Subject: [Cython] hasattr() swallows any exception (Py<3.2) Message-ID: Take a look here: http://bugs.python.org/issue9666 'hasattr' default behaviour should be changed to suppress only AttributeError exceptions. Other should pass through. Should we do something about this in Cython? We currently use PyObject_HasAttr(), but even in Python 3.2 this is not the same as builtins.hasattr(), it swallows any exception (do you think this is a bug in core Python?). I'm inclined to fix the behavior for ALL Python versions to suppress only AttributeError. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Tue Feb 22 22:21:16 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Feb 2011 22:21:16 +0100 Subject: [Cython] hasattr() swallows any exception (Py<3.2) In-Reply-To: References: Message-ID: <4D6428CC.3060608@behnel.de> Lisandro Dalcin, 22.02.2011 21:41: > Take a look here: http://bugs.python.org/issue9666 > > 'hasattr' default behaviour should be changed to suppress only > AttributeError exceptions. Other should pass through. +1, I think I even faintly recall that discussion. What a lengthy thread... http://mail.python.org/pipermail/python-dev/2010-August/103178.html I don't even recall when I last used hasattr(). Given how dumb it is to ask for an attribute, let someone else throw it away for you and then ask again to get it, I don't see much point in using it at all. I can write exception catching code myself, thanks. > Should we do something about this in Cython? We currently use > PyObject_HasAttr(), but even in Python 3.2 this is not the same as > builtins.hasattr(), it swallows any exception (do you think this is a > bug in core Python?). It was at least considered: http://mail.python.org/pipermail/python-dev/2010-August/103203.html > I'm inclined to fix the behavior for ALL Python > versions to suppress only AttributeError. How? Would you implement a hasattr() helper that uses PyObject_GetAttr() and Does The Right Thing? In any case, I personally don't care so much about hasattr(), but I'm +1 for Python compatibility, and +1 for getting the fixed Py3.2 behaviour in all Python versions. There's a reason this was fixed in CPython, it's a pretty clear bug. Stefan From dalcinl at gmail.com Tue Feb 22 23:09:35 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 22 Feb 2011 19:09:35 -0300 Subject: [Cython] hasattr() swallows any exception (Py<3.2) In-Reply-To: <4D6428CC.3060608@behnel.de> References: <4D6428CC.3060608@behnel.de> Message-ID: On 22 February 2011 18:21, Stefan Behnel wrote: > Lisandro Dalcin, 22.02.2011 21:41: >> >> I'm inclined to fix the behavior for ALL Python >> versions to suppress only AttributeError. > > How? Would you implement a hasattr() helper that uses PyObject_GetAttr() and > Does The Right Thing? > Yes, more or less the implementation of builtin_hasattr from bltinmodule.c from Py3.2 > In any case, I personally don't care so much about hasattr(), but I'm +1 for > Python compatibility, and +1 for getting the fixed Py3.2 behaviour in all > Python versions. There's a reason this was fixed in CPython, it's a pretty > clear bug. > OK, I'll go for it. Thanks. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed Feb 23 01:59:18 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 22 Feb 2011 21:59:18 -0300 Subject: [Cython] hasattr() swallows any exception (Py<3.2) In-Reply-To: References: <4D6428CC.3060608@behnel.de> Message-ID: On 22 February 2011 19:09, Lisandro Dalcin wrote: > On 22 February 2011 18:21, Stefan Behnel wrote: >> Lisandro Dalcin, 22.02.2011 21:41: >>> >>> I'm inclined to fix the behavior for ALL Python >>> versions to suppress only AttributeError. >> >> How? Would you implement a hasattr() helper that uses PyObject_GetAttr() and >> Does The Right Thing? >> > > Yes, more or less the implementation of builtin_hasattr from > bltinmodule.c from Py3.2 > >> In any case, I personally don't care so much about hasattr(), but I'm +1 for >> Python compatibility, and +1 for getting the fixed Py3.2 behaviour in all >> Python versions. There's a reason this was fixed in CPython, it's a pretty >> clear bug. >> > > OK, I'll go for it. Thanks. > https://github.com/cython/cython/commit/90120314bcc8bd5a4f71f2629e1065f5c943071c -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From vitja.makarov at gmail.com Wed Feb 23 06:31:45 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 23 Feb 2011 08:31:45 +0300 Subject: [Cython] Control flow graph In-Reply-To: <4D641C26.9070705@behnel.de> References: <4D5A2D6E.9060406@behnel.de> <4D641C26.9070705@behnel.de> Message-ID: 2011/2/22 Stefan Behnel : > Vitja Makarov, 20.02.2011 18:23: >> >> 2011/2/16 Vitja Makarov: >>> >>> Hmm... both python and codespeaks in the thread > > Yes, we should keep it to cython-devel only. Sorry for mixing it up. > > >>> Here is my commit it's mostly broken now but anyway >>> >>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a > > Flow control support is large enough to merit its own module. Not sure how > 'smart' git is here, but you can always keep the history by explicitly > copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated > sections from both files. > Ok. > You are duplicating some code from the type inferencer. We might want to > clean that up at some point. However, given that flow control analysis will > allow us to improve the type inferencer, I think it's best to keep this code > in the FCA part. > Yes, I think it could replace MarkAssignments transform later. Unreachable code could be delete there too. > >> I've update stuff: >> ?- algo for finding definitions >> ?- warnings for uninitialized and may be uninitialised use >> ?- few test cases > > That looks very nice so far. Any idea how well it scales? > "Usually iterative algorithm takes no more then 5 iterations" For ExprNodes.py max number is 15 while avg is about 3 About execution time: ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325, ~10% slow down. -O flag could be introduced but I don't think that's a good idea. Should later try to execute cython compiled code. > >> Trying to compile ParseTreeTransforms.py I've found this for example: >> >> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable >> 'template' may be used uninitialized >> >> ? ? def create_Property(self, entry): >> ? ? ? ? if entry.visibility == 'public': >> ? ? ? ? ? ? if entry.type.is_pyobject: >> ? ? ? ? ? ? ? ? template = self.basic_pyobject_property >> ? ? ? ? ? ? else: >> ? ? ? ? ? ? ? ? template = self.basic_property >> ? ? ? ? elif entry.visibility == 'readonly': >> ? ? ? ? ? ? template = self.basic_property_ro >> ? ? ? ? property = template.substitute({ >> ? ? ? ? ? ? ? ? u"ATTR": ExprNodes.AttributeNode(pos=entry.pos, >> >> obj=ExprNodes.NameNode(pos=entry.pos, name="self"), >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?attribute=entry.name), >> ? ? ? ? ? ? }, pos=entry.pos).stats[0] > > Ok, I guess that code generally works, but it's better to get rid of the > code smell. > Might be used warning should be disabled by default, because algorithm isn't smart enough: a = 1 if (a): b = 1 if (a): print b See also: http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325 -- vitja. From stefan_ml at behnel.de Wed Feb 23 14:42:22 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Feb 2011 14:42:22 +0100 Subject: [Cython] hasattr() swallows any exception (Py<3.2) In-Reply-To: References: <4D6428CC.3060608@behnel.de> Message-ID: <4D650EBE.6000501@behnel.de> Lisandro Dalcin, 23.02.2011 01:59: > On 22 February 2011 19:09, Lisandro Dalcin wrote: >> On 22 February 2011 18:21, Stefan Behnel wrote: >>> Lisandro Dalcin, 22.02.2011 21:41: >>>> >>>> I'm inclined to fix the behavior for ALL Python >>>> versions to suppress only AttributeError. >>> >>> How? Would you implement a hasattr() helper that uses PyObject_GetAttr() and >>> Does The Right Thing? >> >> Yes, more or less the implementation of builtin_hasattr from >> bltinmodule.c from Py3.2 >> >>> In any case, I personally don't care so much about hasattr(), but I'm +1 for >>> Python compatibility, and +1 for getting the fixed Py3.2 behaviour in all >>> Python versions. There's a reason this was fixed in CPython, it's a pretty >>> clear bug. >> >> OK, I'll go for it. Thanks. > > https://github.com/cython/cython/commit/90120314bcc8bd5a4f71f2629e1065f5c943071c Looks good. Thanks. Stefan From dalcinl at gmail.com Wed Feb 23 16:43:54 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 23 Feb 2011 12:43:54 -0300 Subject: [Cython] Py_LIMITED_API Message-ID: Should we try to support Py_LIMITED_API for Py>=3.2? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From robertwb at math.washington.edu Wed Feb 23 20:17:39 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 23 Feb 2011 11:17:39 -0800 Subject: [Cython] Py_LIMITED_API In-Reply-To: References: Message-ID: I think it's worth looking into, but I'm skeptical of the benefits at this point, especially compared to all the other stuff that needs doing. Specifically, I think we'll add complexity to generate more verbose and less efficient code to stick to this. I also think it might be worth keeping in mind but not yet investing a huge amount of effort into, as it may not yet have hit the sweet spot of stability vs. utility, and it may be a lot of effort to work around functions that they decide really should have been part of the core. That being said, I would be curious how much of the non-limited API we use from Cython. If it looks to be an easy or non-invasive change, I'd love to support it. - Robert On Wed, Feb 23, 2011 at 7:43 AM, Lisandro Dalcin wrote: > Should we try to support Py_LIMITED_API for Py>=3.2? > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Wed Feb 23 20:23:04 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Feb 2011 20:23:04 +0100 Subject: [Cython] Py_LIMITED_API In-Reply-To: References: Message-ID: <4D655E98.7070302@behnel.de> Lisandro Dalcin, 23.02.2011 16:43: > Should we try to support Py_LIMITED_API for Py>=3.2? I asked the same question a while ago. I think we should, but it won't be easy. That API is pretty restricted compared to what Cython currently uses, including the internals of builtin types that we use in several places. Even the buffer API is not currently part of the limited API, although this is being worked on. I have some doubts this can be done at C compile time, we may have to add a compiler switch that lets Cython generate different code, and that disables certain optimisations. I agree with Robert that this will likely require more work than we'd currently want to spare, especially given the early state of that API. Stefan From wking at drexel.edu Fri Feb 25 02:42:30 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 24 Feb 2011 20:42:30 -0500 Subject: [Cython] (was: Cython .pxd introspection: listing defined constants) In-Reply-To: <20110222200207.GA14712@tyr.home.net> Message-ID: <20110225014229.GA29938@tyr.home.net> W. Trevor King, 22.02.2011 18:55: > I've been working on a more explicit parser that removes the > ambiguity behind the various visibilities. This will help me ensure > proper impolementation of my cdef-ed enums/structs/..., and make it > easier to update visibility syntax in the future. Take a look and > let me know if you think this is a useful direction to take: The refactoring continues as I'm moving my new binding classes into Symtab. This is, of course, leading to lots of changes, but I've been running the test suite before each commit to make sure I don't go to far astray. Anything I miss will eventually lead to a better test suite ;). I'm currently writing up new versions of most Scope methods that use my classes, and I'll replace the original methods once I've updated all the code that calls them. Next on the list will be the Nodes themselves, at which point I think I'll be positioned to put in the `cdef struct` and whatnot that got this whole thing started. Since there has been a fair amount of churn, I though I'd ask for some more feedback on the general direction I'm headed: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;h=refs/heads/cdef-enums-stucts-and-unions As I said in my previous response to Stefan, I don't really care what the attributes in my classes are called, or how the attributes are partitioned between them. With the current Scope work, I'm leaning towards a single Bindings class that subclasses all of my current bindings, since I have found very few cases where an entire class of nodes only needs one of the bindings, and if we have to pass them all around together, they might as well be a single class. The current names only appear in my new code, so sed & friends will make it easy to tweak things later. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Fri Feb 25 02:57:21 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 24 Feb 2011 17:57:21 -0800 Subject: [Cython] (was: Cython .pxd introspection: listing defined constants) In-Reply-To: <20110225014229.GA29938@tyr.home.net> References: <20110222200207.GA14712@tyr.home.net> <20110225014229.GA29938@tyr.home.net> Message-ID: On Thu, Feb 24, 2011 at 5:42 PM, W. Trevor King wrote: > W. Trevor King, 22.02.2011 18:55: >> I've been working on a more explicit parser that removes the >> ambiguity behind the various visibilities. ?This will help me ensure >> proper impolementation of my cdef-ed enums/structs/..., and make it >> easier to update visibility syntax in the future. ?Take a look and >> let me know if you think this is a useful direction to take: > > The refactoring continues as I'm moving my new binding classes into > Symtab. ?This is, of course, leading to lots of changes, but I've been > running the test suite before each commit to make sure I don't go to > far astray. ?Anything I miss will eventually lead to a better test > suite ;). > > I'm currently writing up new versions of most Scope methods that use > my classes, and I'll replace the original methods once I've updated > all the code that calls them. ?Next on the list will be the Nodes > themselves, at which point I think I'll be positioned to put in the > `cdef struct` and whatnot that got this whole thing started. > > Since there has been a fair amount of churn, I though I'd ask for some > more feedback on the general direction I'm headed: > > ?http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;h=refs/heads/cdef-enums-stucts-and-unions On of my primary motivations to moving to github (or similar) was nicely annotated diffs between branches and the ability to do line-by-line comments. If you could also push to your fork there, that'd be great (otherwise I will ;). - Robert From wking at drexel.edu Fri Feb 25 03:57:07 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 24 Feb 2011 21:57:07 -0500 Subject: [Cython] (was: Cython .pxd introspection: listing defined constants) In-Reply-To: References: <20110222200207.GA14712@tyr.home.net> <20110225014229.GA29938@tyr.home.net> Message-ID: <20110225025005.GA4501@tyr.home.net> On Thu, Feb 24, 2011 at 05:57:21PM -0800, Robert Bradshaw wrote: > On of my primary motivations to moving to github (or similar) was > nicely annotated diffs between branches and the ability to do > line-by-line comments. If you could also push to your fork there, > that'd be great (otherwise I will ;). Ah, Stephan's comment about needing to comment via the mailing list makes more sense now ;). Pushed: https://github.com/wking/cython/tree/cdef-enums-stucts-and-unions -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sat Feb 26 08:11:03 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 25 Feb 2011 23:11:03 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110222200207.GA14712@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> <20110222200207.GA14712@tyr.home.net> Message-ID: On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: > On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote: >> W. Trevor King, 22.02.2011 18:55: >> > I've been working on a more explicit parser that removes the ambiguity >> > behind the various visibilities. ?This will help me ensure proper >> > impolementation of my cdef-ed enums/structs/..., and make it easier to >> > update visibility syntax in the future. ?Take a look and let me know >> > if you think this is a useful direction to take: >> >> First thing that caught my eyes: I hope you do not intend to leave the >> logging usage in the parser. This is seriously performance critical code >> that Cython compiles down to pretty fast C code. Note that you can use a >> debugger and Python's profiling/tracing interface to find out what's >> happening, without code impact. > > I can strip them out afterwards, but it helps me figure out what I've > broken if I shift too much around at the same time. > > I don't know enough about Python's trace module to know if I can turn > on tracing only for functions defined in a single module or not, since > otherwise its hard for me to separate signal from noise. I think you can filter things after the fact. It would also be pretty easy to write a utility that (conditionally) decorates all methods if a flag is set, which we could leave in. (Wouldn't normally be such a big deal, but this is one of the bottlenecks of compilation.) >> Some of the log statements span more than one line, which makes it trickier >> to strip them out with sed&friends (but backing out the initial changeset >> would likely make it simple enough to remove the rest manually). > > Hmm, perhaps I'll condense the logging statements down onto one (long) > line a piece, that will make it easy to comment/uncomment them with > sed/emacs/etc. ?I suppose once Cython can compile the logging module > we could leave them in with reduced overhead ;). > >> Also note that it's best to write runnable tests ("tests/run/"). The >> tests in "tests/compile/" are only compiled and imported. See the >> hacking guide in the wiki. I know you're not there yet with your >> implementation, I'm just mentioning it. > > Thanks for the tip. > >> CtxAttribute is the wrong name, though. ?And the class copy >> implementation gets even more generic than it already was in the >> Ctx. I'm not a big fan of that, especially not in the parser. For >> one, it prevents changing the classes into cdef classes, which had >> long been on my list for Ctx. > > An easy, if uglier, workaround would be to prepend attributes with the > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. > Then the Ctx class could subclass the current CtxAttribute classes > instead of binding instances of each of them. ?That way Ctx would keep > its traditional flat attribute namespace and easy deepcopy, but > eveyone in Nodes, etc. that will use the attributes would become > class-name dependent. I'd be up for flattening this. In particular, changing every "entry.name" to "entry.python_binding.name" seems to be a lot of churn and extra verbiage for not much benefit. The only overlap I see is name and visibility, and keeping name/cname and adding cvisibility would be preferable to me. > The CtxAttribute class is, as its docstring says, just a hook for its > deepcopy method. ?With an alternative deepcopy implementation, > CtxAttribute could be replaced with the standard `object`, so don't > worry too much about its name at this point ;). You mean shallow copy? >> CSource: doesn't sound like quite the right name - it does not describe a C >> source file but information that Cython has about non-Cython things. > > It's a container for attributes that describe the presence and > location of backing C definitions. > > * cdef: "Will there be a backing C defintion? > * extern: "Has someone else already written it?" > * name/namespace: "What did they call it?" > > If you'd rather I called the class something else, I'm certainly > willing to change it. It seems a bit odd to me, but if need be we can rename it later. However, csource and c_binding seem rather redundant to me, but as mentioned above I think it's better just to flatten it all. The changes to parsing look decent to me, but admittedly there's a lot of renaming churn, so I could have missed something. - Robert From stefan_ml at behnel.de Sat Feb 26 08:59:38 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 26 Feb 2011 08:59:38 +0100 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110222200207.GA14712@tyr.home.net> References: <4D5F8C35.4010309@behnel.de> <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> <20110222200207.GA14712@tyr.home.net> Message-ID: <4D68B2EA.5060700@behnel.de> W. Trevor King, 22.02.2011 21:02: > On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote: >> I also doubt that Cython allows you to call an attribute "cdef", you'll >> need to change that. > > It seems to work for me: > >>> import Cython.Compiler.Parsing as P > >>> P.__file__ > 'Cython/Compiler/Parsing.so' > >>> c = P.CSource() > >>> dir(c) > [..., 'cdef', 'deepcopy', 'extern', 'name', 'namespace'] > >>> c.cdef > 0 > >>> c.cdef = 1 > >>> c.cdef > 1 Ah, right. It's actually compiled as .py file, so Cython's syntax doesn't apply here. However, it may apply to other parts of the compiler at some point, so it's better not to use keywords in parts of the source that become globally visible. Stefan From wking at drexel.edu Sat Feb 26 12:48:27 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 26 Feb 2011 06:48:27 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> <20110222200207.GA14712@tyr.home.net> Message-ID: <20110226114826.GA12380@tyr.home.net> On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: > On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: > > An easy, if uglier, workaround would be to prepend attributes with the > > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. > > Then the Ctx class could subclass the current CtxAttribute classes > > instead of binding instances of each of them. That way Ctx would keep > > its traditional flat attribute namespace and easy deepcopy, but > > eveyone in Nodes, etc. that will use the attributes would become > > class-name dependent. > > I'd be up for flattening this. In particular, changing every > "entry.name" to "entry.python_binding.name" seems to be a lot of churn > and extra verbiage for not much benefit. The only overlap I see is > name and visibility, and keeping name/cname and adding cvisibility > would be preferable to me. That works for me, but I see possible ambiguity in cname. From the "external C code" docs: The other way is to use a C name specification to give different Cython and C names to the C function. Suppose, for example, that you want to wrap an external function called eject_tomato(). If you declare it as: cdef extern void c_eject_tomato "eject_tomato" (float speed) then its name inside the Cython module will be c_eject_tomato, whereas its name in C will be eject_tomato. In this case I was eventually going to use c_source.name = eject_tomato c_source.namespace = ... c_binding.name = c_eject_tomato c_binding.namespace = ... python_binding.name = eject_tomato I'm not sure how Cython handles name/cname with this case at the moment, but it seems like there are two cnames to keep track of. > > The CtxAttribute class is, as its docstring says, just a hook for its > > deepcopy method. With an alternative deepcopy implementation, > > CtxAttribute could be replaced with the standard `object`, so don't > > worry too much about its name at this point ;). > > You mean shallow copy? I meant deepcopy, since that seems to be the point of Ctx cloning. At the moment, however, Ctx deep copies only require Binding shallow copies. If someone crazy wanted to add mutable attrubites to binding classes, the Binding deepcopy code would have had to be adjusted. None of this matters though, if we move back to a flat Ctx attribute space. > >> CSource: doesn't sound like quite the right name - it does not describe a C > >> source file but information that Cython has about non-Cython things. > > > > It's a container for attributes that describe the presence and > > location of backing C definitions. > > > > * cdef: "Will there be a backing C defintion? > > * extern: "Has someone else already written it?" > > * name/namespace: "What did they call it?" > > > > If you'd rather I called the class something else, I'm certainly > > willing to change it. > > It seems a bit odd to me, but if need be we can rename it later. > However, csource and c_binding seem rather redundant to me, but as > mentioned above I think it's better just to flatten it all. > > The changes to parsing look decent to me, but admittedly there's a lot > of renaming churn, so I could have missed something. I'll go back and revert out the name changes if you'll confirm that there is only one meaning for cname. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From wking at drexel.edu Sat Feb 26 14:08:41 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 26 Feb 2011 08:08:41 -0500 Subject: [Cython] Expected errors of tests/errors/cdef_members_T517.pxd Message-ID: <20110226130841.GA21458@tyr.home.net> I'm splitting Symtab visibilities into explicit C and Python visibilities, but am having trouble reproducing the expected error messages for cdef_members_T517: $ python runtests.py cdef_members_T517 Python 2.6.6 (r266:84292, Dec 8 2010, 09:53:33) [GCC 4.4.4] Running tests against Cython 0.14.1+ === Expected errors: === 5:24: C attribute of type 'VoidP' cannot be accessed from Python 5:24: Cannot convert 'VoidP' to Python object 6:24: C attribute of type 'VoidP' cannot be accessed from Python 6:24: Cannot convert 'VoidP' to Python object 6:24: Cannot convert Python object to 'VoidP' 14:22: C attribute of type 'Foo' cannot be accessed from Python 14:22: Cannot convert Python object to 'Foo' === Got errors: === 5:24: C attribute of type 'VoidP' cannot be accessed from Python 5:24: Cannot convert 'VoidP' to Python object 6:24: C attribute of type 'VoidP' cannot be accessed from Python 6:24: Cannot convert 'VoidP' to Python object 6:24: Cannot convert Python object to 'VoidP' 14:22: Cannot convert Python object to 'Foo' So my code is not raising: 14:22: C attribute of type 'Foo' cannot be accessed from Python The relevant lines from tests/errors/cdef_members_T517.pxd are: $ grep -n ^ tests/errors/cdef_members_T517.pyx ... 8:ctypedef struct Foo: 9: int i 10: 11:cdef class Bar: 12: cdef Foo foo0 13: cdef readonly Foo foo2 14: cdef public Foo foo1 15: pass ... I see two options: 1) Foo attributes are readable from Python, in which case the expected errors should be changed to match mine. 2) Foo attributes are not readable from Python, in which case the expected errors should be extended with 13:22: C attribute of type 'Foo' cannot be accessed from Python and probably also also 14:22: Cannot convert 'Foo' to Python object I think (2) is more reasonable, until I get my public Python enums, stucts, and unions branch working, after which it would switch to (1) if the Foo struct was declared with readonly (vs. private) Python visibility. You can't alter the struct definition from Python, so a public Python visibility doesn't really make sense for the struct itself. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sat Feb 26 19:01:43 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 26 Feb 2011 10:01:43 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110226114826.GA12380@tyr.home.net> References: <20110219192202.GA14142@tyr.home.net> <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> <20110222200207.GA14712@tyr.home.net> <20110226114826.GA12380@tyr.home.net> Message-ID: On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King wrote: > On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: >> On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: >> > An easy, if uglier, workaround would be to prepend attributes with the >> > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. >> > Then the Ctx class could subclass the current CtxAttribute classes >> > instead of binding instances of each of them. ?That way Ctx would keep >> > its traditional flat attribute namespace and easy deepcopy, but >> > eveyone in Nodes, etc. that will use the attributes would become >> > class-name dependent. >> >> I'd be up for flattening this. In particular, changing every >> "entry.name" to "entry.python_binding.name" seems to be a lot of churn >> and extra verbiage for not much benefit. The only overlap I see is >> name and visibility, and keeping name/cname and adding cvisibility >> would be preferable to me. > > That works for me, but I see possible ambiguity in cname. ?From the > "external C code" docs: > > ? ?The other way is to use a C name specification to give different > ? ?Cython and C names to the C function. Suppose, for example, that > ? ?you want to wrap an external function called eject_tomato(). If > ? ?you declare it as: > > ? ?cdef extern void c_eject_tomato "eject_tomato" (float speed) > > ? ?then its name inside the Cython module will be c_eject_tomato, > ? ?whereas its name in C will be eject_tomato. > > In this case I was eventually going to use > > ? ?c_source.name = eject_tomato > ? ?c_source.namespace = ... > ? ? ? ? ? ? ? ?c_binding.name = c_eject_tomato > ? ?c_binding.namespace = ... > ? ? ? ? ? ? ? ?python_binding.name = eject_tomato > > I'm not sure how Cython handles name/cname with this case at the > moment, but it seems like there are two cnames to keep track of. In this case, cname is "eject_tomato" (which actually gets emitted in the C source file to use it) and name (in both Python and Cython namespace) is "c_eject_tomato." You can think of it as name being what the user sees, and cname being what the C compiler sees. >> > The CtxAttribute class is, as its docstring says, just a hook for its >> > deepcopy method. ?With an alternative deepcopy implementation, >> > CtxAttribute could be replaced with the standard `object`, so don't >> > worry too much about its name at this point ;). >> >> You mean shallow copy? > > I meant deepcopy, since that seems to be the point of Ctx cloning. ?At > the moment, however, Ctx deep copies only require Binding shallow > copies. ?If someone crazy wanted to add mutable attrubites to binding > classes, the Binding deepcopy code would have had to be adjusted. > None of this matters though, if we move back to a flat Ctx attribute > space. OK. As an aside, do you know about the copy.deepcopy() method? >> >> CSource: doesn't sound like quite the right name - it does not describe a C >> >> source file but information that Cython has about non-Cython things. >> > >> > It's a container for attributes that describe the presence and >> > location of backing C definitions. >> > >> > * cdef: "Will there be a backing C defintion? >> > * extern: "Has someone else already written it?" >> > * name/namespace: "What did they call it?" >> > >> > If you'd rather I called the class something else, I'm certainly >> > willing to change it. >> >> It seems a bit odd to me, but if need be we can rename it later. >> However, csource and c_binding seem rather redundant to me, but as >> mentioned above I think it's better just to flatten it all. >> >> The changes to parsing look decent to me, but admittedly there's a lot >> of renaming churn, so I could have missed something. > > I'll go back and revert out the name changes if you'll confirm that > there is only one meaning for cname. Thanks. BTW, I pushed https://github.com/cython/cython/commit/3552114e1b2a21bf2f81f451d50cd48934ea4eb4 which starts to do some of the back-end implementation. - Robert From robertwb at math.washington.edu Sat Feb 26 19:15:38 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 26 Feb 2011 10:15:38 -0800 Subject: [Cython] Expected errors of tests/errors/cdef_members_T517.pxd In-Reply-To: <20110226130841.GA21458@tyr.home.net> References: <20110226130841.GA21458@tyr.home.net> Message-ID: On Sat, Feb 26, 2011 at 5:08 AM, W. Trevor King wrote: > I'm splitting Symtab visibilities into explicit C and Python > visibilities, but am having trouble reproducing the expected error > messages for cdef_members_T517: > > ? ?$ python runtests.py cdef_members_T517 > ? ?Python 2.6.6 (r266:84292, Dec ?8 2010, 09:53:33) > ? ?[GCC 4.4.4] > > ? ?Running tests against Cython 0.14.1+ > > ? ?=== Expected errors: === > ? ?5:24: C attribute of type 'VoidP' cannot be accessed from Python > ? ?5:24: Cannot convert 'VoidP' to Python object > ? ?6:24: C attribute of type 'VoidP' cannot be accessed from Python > ? ?6:24: Cannot convert 'VoidP' to Python object > ? ?6:24: Cannot convert Python object to 'VoidP' > ? ?14:22: C attribute of type 'Foo' cannot be accessed from Python > ? ?14:22: Cannot convert Python object to 'Foo' > > > ? ?=== Got errors: === > ? ?5:24: C attribute of type 'VoidP' cannot be accessed from Python > ? ?5:24: Cannot convert 'VoidP' to Python object > ? ?6:24: C attribute of type 'VoidP' cannot be accessed from Python > ? ?6:24: Cannot convert 'VoidP' to Python object > ? ?6:24: Cannot convert Python object to 'VoidP' > ? ?14:22: Cannot convert Python object to 'Foo' > > So my code is not raising: > > ? ?14:22: C attribute of type 'Foo' cannot be accessed from Python > > The relevant lines from tests/errors/cdef_members_T517.pxd are: > > ? ?$ grep -n ^ tests/errors/cdef_members_T517.pyx > ? ?... > ? ?8:ctypedef struct Foo: > ? ?9: ? ?int i > ? ?10: > ? ?11:cdef class Bar: > ? ?12: ? ?cdef ? ? ? ? ?Foo foo0 > ? ?13: ? ?cdef readonly Foo foo2 > ? ?14: ? ?cdef public ? Foo foo1 > ? ?15: ? ?pass > ? ?... > > I see two options: > > 1) Foo attributes are readable from Python, in which case the expected > ? errors should be changed to match mine. Foo attributes are readable from Python, it converts the struct to a tuple. > 2) Foo attributes are not readable from Python, in which case the > ? expected errors should be extended with > > ? ?13:22: C attribute of type 'Foo' cannot be accessed from Python > > ? and probably also also > > ? ?14:22: Cannot convert 'Foo' to Python object > > I think (2) is more reasonable, until I get my public Python enums, > stucts, and unions branch working, after which it would switch to (1) > if the Foo struct was declared with readonly (vs. private) Python > visibility. ?You can't alter the struct definition from Python, so a > public Python visibility doesn't really make sense for the struct > itself. The visibility refers to the assignability of the member from Python, which makes total sense if there is a object -> Foo conversion. I thought I had implemented this, but I guess now. Once we have wrapping classes this makes total sense, so one can do b = Bar() b.foo1 = Foo(i=100) What's less clear is the behavior of b = Bar() b.foo1 = Foo(i=1) b.foo1.i = 100 print b.foo1 saved = b.foo1 del b print saved - Robert From wking at drexel.edu Sun Feb 27 03:14:12 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 26 Feb 2011 21:14:12 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> <20110222200207.GA14712@tyr.home.net> <20110226114826.GA12380@tyr.home.net> Message-ID: <20110227021409.GA3698@tyr.home.net> On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote: > On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King wrote: > > On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: > >> On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: > >> > An easy, if uglier, workaround would be to prepend attributes with the > >> > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. > >> > Then the Ctx class could subclass the current CtxAttribute classes > >> > instead of binding instances of each of them. That way Ctx would keep > >> > its traditional flat attribute namespace and easy deepcopy, but > >> > eveyone in Nodes, etc. that will use the attributes would become > >> > class-name dependent. > >> > >> I'd be up for flattening this. In particular, changing every > >> "entry.name" to "entry.python_binding.name" seems to be a lot of churn > >> and extra verbiage for not much benefit. The only overlap I see is > >> name and visibility, and keeping name/cname and adding cvisibility > >> would be preferable to me. > > > > That works for me, but I see possible ambiguity in cname. From the > > "external C code" docs: > > > > The other way is to use a C name specification to give different > > Cython and C names to the C function. Suppose, for example, that > > you want to wrap an external function called eject_tomato(). If > > you declare it as: > > > > cdef extern void c_eject_tomato "eject_tomato" (float speed) > > > > then its name inside the Cython module will be c_eject_tomato, > > whereas its name in C will be eject_tomato. > > > > In this case I was eventually going to use > > > > c_source.name = eject_tomato > > c_source.namespace = ... > > c_binding.name = c_eject_tomato > > c_binding.namespace = ... > > python_binding.name = eject_tomato > > > > I'm not sure how Cython handles name/cname with this case at the > > moment, but it seems like there are two cnames to keep track of. > > In this case, cname is "eject_tomato" (which actually gets emitted in > the C source file to use it) and name (in both Python and Cython > namespace) is "c_eject_tomato." What is the "Cython namespace"? Is that what you get when you cimport something (vs. the Python namespace being what you get when you import something). If so, then that sounds like just two names, so name/cname it is. > >> > The CtxAttribute class is, as its docstring says, just a hook for its > >> > deepcopy method. With an alternative deepcopy implementation, > >> > CtxAttribute could be replaced with the standard `object`, so don't > >> > worry too much about its name at this point ;). > >> > >> You mean shallow copy? > > > > I meant deepcopy, since that seems to be the point of Ctx cloning. At > > the moment, however, Ctx deep copies only require Binding shallow > > copies. If someone crazy wanted to add mutable attrubites to binding > > classes, the Binding deepcopy code would have had to be adjusted. > > None of this matters though, if we move back to a flat Ctx attribute > > space. > > OK. As an aside, do you know about the copy.deepcopy() method? Yup, but I was trying to avoid pulling in non-Cython dependencies since you guys seem to have gone through a lot of trouble to make the Parsing module fast in C. > >> >> CSource: doesn't sound like quite the right name - it does not describe a C > >> >> source file but information that Cython has about non-Cython things. > >> > > >> > It's a container for attributes that describe the presence and > >> > location of backing C definitions. > >> > > >> > * cdef: "Will there be a backing C defintion? > >> > * extern: "Has someone else already written it?" > >> > * name/namespace: "What did they call it?" > >> > > >> > If you'd rather I called the class something else, I'm certainly > >> > willing to change it. > >> > >> It seems a bit odd to me, but if need be we can rename it later. > >> However, csource and c_binding seem rather redundant to me, but as > >> mentioned above I think it's better just to flatten it all. > >> > >> The changes to parsing look decent to me, but admittedly there's a lot > >> of renaming churn, so I could have missed something. > > > > I'll go back and revert out the name changes if you'll confirm that > > there is only one meaning for cname. > > Thanks. > > BTW, I pushed https://github.com/cython/cython/commit/3552114e1b2a21bf2f81f451d50cd48934ea4eb4 > which starts to do some of the back-end implementation. Ok, I'll merge those once I get the visibility kinks worked out of Symtab. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From wking at drexel.edu Sun Feb 27 03:34:03 2011 From: wking at drexel.edu (W. Trevor King) Date: Sat, 26 Feb 2011 21:34:03 -0500 Subject: [Cython] Expected errors of tests/errors/cdef_members_T517.pxd In-Reply-To: References: <20110226130841.GA21458@tyr.home.net> Message-ID: <20110227023403.GB3698@tyr.home.net> On Sat, Feb 26, 2011 at 10:15:38AM -0800, Robert Bradshaw wrote: > On Sat, Feb 26, 2011 at 5:08 AM, W. Trevor King wrote: > > The relevant lines from tests/errors/cdef_members_T517.pxd are: > > > > $ grep -n ^ tests/errors/cdef_members_T517.pyx > > ... > > 8:ctypedef struct Foo: > > 9: int i > > 10: > > 11:cdef class Bar: > > 12: cdef Foo foo0 > > 13: cdef readonly Foo foo2 > > 14: cdef public Foo foo1 > > 15: pass > > ... > > ... > > What's less clear is the behavior of > > b = Bar() > b.foo1 = Foo(i=1) > b.foo1.i = 100 > print b.foo1 > saved = b.foo1 > del b > print saved Is the problem choosing between: 1) Copy struct data into its own PyObject and let Python manage the memory. 2) Pass around a pointer to Cython-managed struct data. I suppose I don't really know enough about how Cython handles this yet to really see the problem or be able to help with a solution ;). -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sun Feb 27 04:11:48 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 26 Feb 2011 19:11:48 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110227021409.GA3698@tyr.home.net> References: <20110219214515.GB25132@tyr.home.net> <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> <20110222200207.GA14712@tyr.home.net> <20110226114826.GA12380@tyr.home.net> <20110227021409.GA3698@tyr.home.net> Message-ID: On Sat, Feb 26, 2011 at 6:14 PM, W. Trevor King wrote: > On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote: >> On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King wrote: >> > On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: >> >> On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: >> >> > An easy, if uglier, workaround would be to prepend attributes with the >> >> > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. >> >> > Then the Ctx class could subclass the current CtxAttribute classes >> >> > instead of binding instances of each of them. ?That way Ctx would keep >> >> > its traditional flat attribute namespace and easy deepcopy, but >> >> > eveyone in Nodes, etc. that will use the attributes would become >> >> > class-name dependent. >> >> >> >> I'd be up for flattening this. In particular, changing every >> >> "entry.name" to "entry.python_binding.name" seems to be a lot of churn >> >> and extra verbiage for not much benefit. The only overlap I see is >> >> name and visibility, and keeping name/cname and adding cvisibility >> >> would be preferable to me. >> > >> > That works for me, but I see possible ambiguity in cname. ?From the >> > "external C code" docs: >> > >> > ? ?The other way is to use a C name specification to give different >> > ? ?Cython and C names to the C function. Suppose, for example, that >> > ? ?you want to wrap an external function called eject_tomato(). If >> > ? ?you declare it as: >> > >> > ? ?cdef extern void c_eject_tomato "eject_tomato" (float speed) >> > >> > ? ?then its name inside the Cython module will be c_eject_tomato, >> > ? ?whereas its name in C will be eject_tomato. >> > >> > In this case I was eventually going to use >> > >> > ? ?c_source.name = eject_tomato >> > ? ?c_source.namespace = ... >> > ? ? ? ? ? ? ? ?c_binding.name = c_eject_tomato >> > ? ?c_binding.namespace = ... >> > ? ? ? ? ? ? ? ?python_binding.name = eject_tomato >> > >> > I'm not sure how Cython handles name/cname with this case at the >> > moment, but it seems like there are two cnames to keep track of. >> >> In this case, cname is "eject_tomato" (which actually gets emitted in >> the C source file to use it) and name (in both Python and Cython >> namespace) is "c_eject_tomato." > > What is the "Cython namespace"? ?Is that what you get when you cimport > something (vs. the Python namespace being what you get when you import > something). Yes, that would be a good way to understand it. > If so, then that sounds like just two names, so > name/cname it is. Anything that exists in the intersection of the Cython and Python namespace has the same name in both places--the user-visible one, thus name suffices for both. (Also, the Cython namespace is a superset of the Python namespace, possibly with a semantically equivalent but more optimized backing for some objects, e.g. dispatching to C methods directly for builtins.) >> >> > The CtxAttribute class is, as its docstring says, just a hook for its >> >> > deepcopy method. ?With an alternative deepcopy implementation, >> >> > CtxAttribute could be replaced with the standard `object`, so don't >> >> > worry too much about its name at this point ;). >> >> >> >> You mean shallow copy? >> > >> > I meant deepcopy, since that seems to be the point of Ctx cloning. ?At >> > the moment, however, Ctx deep copies only require Binding shallow >> > copies. ?If someone crazy wanted to add mutable attrubites to binding >> > classes, the Binding deepcopy code would have had to be adjusted. >> > None of this matters though, if we move back to a flat Ctx attribute >> > space. >> >> OK. As an aside, do you know about the copy.deepcopy() method? > > Yup, but I was trying to avoid pulling in non-Cython dependencies > since you guys seem to have gone through a lot of trouble to make the > Parsing module fast in C. Ah, OK. It's not always obvious without looking what from the standard library is fast and what is not, but this doesn't look particularly optimized. - Robert From robertwb at math.washington.edu Sun Feb 27 04:16:37 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 26 Feb 2011 19:16:37 -0800 Subject: [Cython] Expected errors of tests/errors/cdef_members_T517.pxd In-Reply-To: <20110227023403.GB3698@tyr.home.net> References: <20110226130841.GA21458@tyr.home.net> <20110227023403.GB3698@tyr.home.net> Message-ID: On Sat, Feb 26, 2011 at 6:34 PM, W. Trevor King wrote: > On Sat, Feb 26, 2011 at 10:15:38AM -0800, Robert Bradshaw wrote: >> On Sat, Feb 26, 2011 at 5:08 AM, W. Trevor King wrote: >> > The relevant lines from tests/errors/cdef_members_T517.pxd are: >> > >> > ? ?$ grep -n ^ tests/errors/cdef_members_T517.pyx >> > ? ?... >> > ? ?8:ctypedef struct Foo: >> > ? ?9: ? ?int i >> > ? ?10: >> > ? ?11:cdef class Bar: >> > ? ?12: ? ?cdef ? ? ? ? ?Foo foo0 >> > ? ?13: ? ?cdef readonly Foo foo2 >> > ? ?14: ? ?cdef public ? Foo foo1 >> > ? ?15: ? ?pass >> > ? ?... >> >> ... >> >> What's less clear is the behavior of >> >> ? ?b = Bar() >> ? ?b.foo1 = Foo(i=1) >> ? ?b.foo1.i = 100 >> ? ?print b.foo1 >> ? ?saved = b.foo1 >> ? ?del b >> ? ?print saved > > Is the problem choosing between: > > 1) Copy struct data into its own PyObject and let Python manage the > ? memory. > 2) Pass around a pointer to Cython-managed struct data. Yes, exactly. > I suppose I don't really know enough about how Cython handles this yet > to really see the problem or be able to help with a solution ;). Cython doesn't handle this yet--that's what we get to decide. Another option would be to use weakrefs to implement copy-on-owner-destruction, but that has even subtler corner semantics. ? ?b = Bar() ? ?b.foo1 = Foo(i=1) ? ?first = b.foo1 second = b.foo1 first.i = 100 print second ? ?del b first.i = 200 print second I'm leaning towards (1) for ease and efficiency of implementation and it's easier to explain in all cases. - Robert From dalcinl at gmail.com Mon Feb 28 17:33:16 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 28 Feb 2011 13:33:16 -0300 Subject: [Cython] adding support for __dict__ in extension types In-Reply-To: <5E596740-6D0C-4AB9-9149-A28698DA1789@math.washington.edu> References: <5E596740-6D0C-4AB9-9149-A28698DA1789@math.washington.edu> Message-ID: Bringing up this old post... On 21 June 2010 15:41, Robert Bradshaw wrote: > On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: > >> If we special case a __dict__ attribute in extension types, i.e: >> >> cdef class Foo: >> ? ?cdef dict __dict__ >> >> and fill type->tp_dictoffset, then we can support __dict__ in >> extension types. >> >> What do you think? > > Sounds like a good idea to me. Note that we check tp_dictoffset for > fast dispatching for cpdef methods (which would be correct as a dict > lookup *would* be needed if __dict__ is available). > I still have this patch lying around in my disk. I remember Stefan had some objections. For example, when the user ask for __dict__, a new dict is unconditionally created (in CPython, type dict are allocated on-demand). I propose to get this patch pushed now, and optimize later (however, I really don't know how to safely implement this optimization). -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 -------------- next part -------------- A non-text attachment was scrubbed... Name: exttypedict.diff Type: text/x-patch Size: 7985 bytes Desc: not available URL: From stefan_ml at behnel.de Mon Feb 28 18:02:09 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 28 Feb 2011 18:02:09 +0100 Subject: [Cython] adding support for __dict__ in extension types In-Reply-To: References: <5E596740-6D0C-4AB9-9149-A28698DA1789@math.washington.edu> Message-ID: <4D6BD511.1050408@behnel.de> Lisandro Dalcin, 28.02.2011 17:33: > Bringing up this old post... > > On 21 June 2010 15:41, Robert Bradshaw wrote: >> On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: >> >>> If we special case a __dict__ attribute in extension types, i.e: >>> >>> cdef class Foo: >>> cdef dict __dict__ >>> >>> and fill type->tp_dictoffset, then we can support __dict__ in >>> extension types. >>> >>> What do you think? >> >> Sounds like a good idea to me. Note that we check tp_dictoffset for >> fast dispatching for cpdef methods (which would be correct as a dict >> lookup *would* be needed if __dict__ is available). >> > > I still have this patch lying around in my disk. I remember Stefan had > some objections. For example, when the user ask for __dict__, a new > dict is unconditionally created (in CPython, type dict are allocated > on-demand). I propose to get this patch pushed now, and optimize > later (however, I really don't know how to safely implement this > optimization). For reference: http://thread.gmane.org/gmane.comp.python.cython.devel/10189 Stefan From robertwb at math.washington.edu Mon Feb 28 19:09:34 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Feb 2011 10:09:34 -0800 Subject: [Cython] adding support for __dict__ in extension types In-Reply-To: References: <5E596740-6D0C-4AB9-9149-A28698DA1789@math.washington.edu> Message-ID: On Mon, Feb 28, 2011 at 8:33 AM, Lisandro Dalcin wrote: > Bringing up this old post... > > On 21 June 2010 15:41, Robert Bradshaw wrote: >> On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: >> >>> If we special case a __dict__ attribute in extension types, i.e: >>> >>> cdef class Foo: >>> ? ?cdef dict __dict__ >>> >>> and fill type->tp_dictoffset, then we can support __dict__ in >>> extension types. >>> >>> What do you think? >> >> Sounds like a good idea to me. Note that we check tp_dictoffset for >> fast dispatching for cpdef methods (which would be correct as a dict >> lookup *would* be needed if __dict__ is available). >> > > I still have this patch lying around in my disk. I remember Stefan had > some objections. For example, when the user ask for __dict__, a new > dict is unconditionally created (in CPython, type dict are allocated > on-demand). ?I propose to get this patch pushed now, and optimize > later (however, I really don't know how to safely implement this > optimization). Note there's also the issue of cpdef methods--if the instance has a __dict__ then a dict lookup must be performed for every method call (to make sure it's not overridden). - Robert From dalcinl at gmail.com Mon Feb 28 19:45:44 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 28 Feb 2011 15:45:44 -0300 Subject: [Cython] adding support for __dict__ in extension types In-Reply-To: References: <5E596740-6D0C-4AB9-9149-A28698DA1789@math.washington.edu> Message-ID: On 28 February 2011 15:09, Robert Bradshaw wrote: > On Mon, Feb 28, 2011 at 8:33 AM, Lisandro Dalcin wrote: >> Bringing up this old post... >> >> On 21 June 2010 15:41, Robert Bradshaw wrote: >>> On Jun 17, 2010, at 9:31 AM, Lisandro Dalcin wrote: >>> >>>> If we special case a __dict__ attribute in extension types, i.e: >>>> >>>> cdef class Foo: >>>> ? ?cdef dict __dict__ >>>> >>>> and fill type->tp_dictoffset, then we can support __dict__ in >>>> extension types. >>>> >>>> What do you think? >>> >>> Sounds like a good idea to me. Note that we check tp_dictoffset for >>> fast dispatching for cpdef methods (which would be correct as a dict >>> lookup *would* be needed if __dict__ is available). >>> >> >> I still have this patch lying around in my disk. I remember Stefan had >> some objections. For example, when the user ask for __dict__, a new >> dict is unconditionally created (in CPython, type dict are allocated >> on-demand). ?I propose to get this patch pushed now, and optimize >> later (however, I really don't know how to safely implement this >> optimization). > > Note there's also the issue of cpdef methods--if the instance has a > __dict__ then a dict lookup must be performed for every method call > (to make sure it's not overridden). > But if the type do have a __dict__, then tp_dictoffset will be filled (this is in my patch), and cpdef methods should always go the dict lookup... Am I missing something? Now that you mention it, my patch should include tests for all this. I'll work on that. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From thomas.e.keller at gmail.com Mon Feb 28 21:10:09 2011 From: thomas.e.keller at gmail.com (Thomas Keller) Date: Mon, 28 Feb 2011 14:10:09 -0600 Subject: [Cython] lurker thanks ya'll for all the awesome new stuff you're tinkering on Message-ID: I don't have the time or knowledge to contribute to Cython right now, but I just want to thank everybody for the hard work and time they've spent on the project. I do usually read the emails that come from this listserv, and it seems like things are progressing rapidly! I sincerely hope everyone has a great day. Cheers, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitja.makarov at gmail.com Mon Feb 28 21:30:15 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 28 Feb 2011 23:30:15 +0300 Subject: [Cython] Control flow graph In-Reply-To: References: <4D5A2D6E.9060406@behnel.de> <4D641C26.9070705@behnel.de> Message-ID: 2011/2/23 Vitja Makarov : > 2011/2/23 Vitja Makarov : >> 2011/2/22 Stefan Behnel : >>> Vitja Makarov, 20.02.2011 18:23: >>>> >>>> 2011/2/16 Vitja Makarov: >>>>> >>>>> Hmm... both python and codespeaks in the thread >>> >>> Yes, we should keep it to cython-devel only. Sorry for mixing it up. >>> >>> >>>>> Here is my commit it's mostly broken now but anyway >>>>> >>>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a >>> >>> Flow control support is large enough to merit its own module. Not sure how >>> 'smart' git is here, but you can always keep the history by explicitly >>> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated >>> sections from both files. >>> >> >> Ok. >> > > Oops there is no copy command in git. > > >>> You are duplicating some code from the type inferencer. We might want to >>> clean that up at some point. However, given that flow control analysis will >>> allow us to improve the type inferencer, I think it's best to keep this code >>> in the FCA part. >>> >> >> Yes, I think it could replace MarkAssignments transform later. >> Unreachable code could be delete there too. >> >>> >>>> I've update stuff: >>>> ?- algo for finding definitions >>>> ?- warnings for uninitialized and may be uninitialised use >>>> ?- few test cases >>> >>> That looks very nice so far. Any idea how well it scales? >>> >> >> "Usually iterative algorithm takes no more then 5 iterations" >> >> For ExprNodes.py max number is 15 while avg is about 3 >> >> About execution time: >> >> ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325, >> ~10% slow down. >> -O flag could be introduced but I don't think that's a good idea. >> >> Should later try to execute cython compiled code. >> >>> >>>> Trying to compile ParseTreeTransforms.py I've found this for example: >>>> >>>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable >>>> 'template' may be used uninitialized >>>> >>>> ? ? def create_Property(self, entry): >>>> ? ? ? ? if entry.visibility == 'public': >>>> ? ? ? ? ? ? if entry.type.is_pyobject: >>>> ? ? ? ? ? ? ? ? template = self.basic_pyobject_property >>>> ? ? ? ? ? ? else: >>>> ? ? ? ? ? ? ? ? template = self.basic_property >>>> ? ? ? ? elif entry.visibility == 'readonly': >>>> ? ? ? ? ? ? template = self.basic_property_ro >>>> ? ? ? ? property = template.substitute({ >>>> ? ? ? ? ? ? ? ? u"ATTR": ExprNodes.AttributeNode(pos=entry.pos, >>>> >>>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"), >>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?attribute=entry.name), >>>> ? ? ? ? ? ? }, pos=entry.pos).stats[0] >>> >>> Ok, I guess that code generally works, but it's better to get rid of the >>> code smell. >>> >> >> Might be used warning should be disabled by default, because algorithm >> isn't smart enough: >> >> a = 1 >> if (a): b = 1 >> if (a): print b >> >> See also: >> http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325 >> I've updated things: - Now it passes all the tests (some tests are run with remove_unreachable=False) - Control flow graph support all the control statements (hope so) - Analyse PyClassDefNode (now only inner, need some more work) - Now we have the following optional warnings: 1. uninitialized variable use [default on] 2. possibly uninitialized use [-Wextra] 3. unreachable code [default on] (this needs rework, unreachable code detection could be better handled inside CreateControlFlowGraph) 4. unused variable [-Wextra] 5. unused argument [-X warn.unused_arg=True] 6. unused result [-X warn_unused_result=True] - Optional dot graph output: $ cython foo.pyx -X control_flow.dot_output=foo.dot $ dot -Tpng -o foo.png foo.dot -- vitja.