From marko.loparic at gmail.com Tue Mar 1 10:42:15 2011 From: marko.loparic at gmail.com (Marko Loparic) Date: Tue, 1 Mar 2011 10:42:15 +0100 Subject: [Cython] lurker thanks ya'll for all the awesome new stuff you're tinkering on In-Reply-To: References: Message-ID: On 28 February 2011 21:10, Thomas Keller wrote: > I don't have the time or knowledge to contribute to Cython right now, but I just want to thank everybody for the hard > work and time they've spent on the project. +1. I wish you lots of fun for the Cython workshop! From robertwb at math.washington.edu Wed Mar 2 03:45:46 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 1 Mar 2011 18:45:46 -0800 Subject: [Cython] lurker thanks ya'll for all the awesome new stuff you're tinkering on In-Reply-To: References: Message-ID: Thanks both of you for the encouragement. Glad Cython is well liked and well used. On Tue, Mar 1, 2011 at 1:42 AM, Marko Loparic wrote: > On 28 February 2011 21:10, Thomas Keller wrote: >> I don't have the time or knowledge to contribute to Cython right now, but I just want to thank everybody for the hard >> work and time they've spent on the project. > > +1. I wish you lots of fun for the Cython workshop! > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From dagss at student.matnat.uio.no Wed Mar 2 11:20:47 2011 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Mar 2011 11:20:47 +0100 Subject: [Cython] Multiple modules in one compilation unit Message-ID: <4D6E19FF.1010002@student.matnat.uio.no> This has been raised earlier, but I don't think there was such a demonstrative use-case as what I have now. Fwrap is suppose to be able to wrap Fortran "modules", which is essentially a namespace mechanism. It makes sense to convert the namespace to Python by creating one Cython pyx file per Fortran module. However, if you are wrapping a Fortran library this way, you suddenly have lots of opportunity to mess up the build: - If you build the Fortran code as a static library (rather common...), then each pyx file will have their own copy. This will link successfully but likely have a rather poor effect. - If you link each Cython file with only the corresponding Fortran file, things won't work (likely to get missing symbols from cross-module calls Fortran-side). Yes, linking each Cython file to the same shared Fortran library should work. Still, this seems dangerous. Options: a) Simply make sure to link with shared versions of Fortran libraries ("this is a documentation problem") b) Use some other namespace mechanism in the same pyx (classes with static methods...) c) Somehow provide more than one module in the same compilation unit. Again, this requires the build to work correctly, but seems less dangerous, and also has the advantage of *allowing* static linking of the Fortran library, if one wants to. But is something like this possible? Could one implement cython -o assembly.c file1.pyx file2.pyx file3.pyx ...where assembly.so would contain three Python modules? (initfile1, initfile2, and so on...) Dag Sverre From stefan_ml at behnel.de Wed Mar 2 11:48:25 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 02 Mar 2011 11:48:25 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E19FF.1010002@student.matnat.uio.no> References: <4D6E19FF.1010002@student.matnat.uio.no> Message-ID: <4D6E2079.9000102@behnel.de> Dag Sverre Seljebotn, 02.03.2011 11:20: > c) Somehow provide more than one module in the same compilation unit. > Again, this requires the build to work correctly, but seems less dangerous, > and also has the advantage of *allowing* static linking of the Fortran > library, if one wants to. > > But is something like this possible? Could one implement > > cython -o assembly.c file1.pyx file2.pyx file3.pyx > > ...where assembly.so would contain three Python modules? (initfile1, > initfile2, and so on...) Can't currently work because the three modules would define the same static C names. This could be fixed as part of the PEP 3121 implementation: http://trac.cython.org/cython_trac/ticket/173 Or it could be worked around by overriding the prefixes in Naming.py (which sounds ugly). Generally speaking, I'm -1 on this idea. I don't see a real use case, and you're saying yourself that this isn't required to make your Fortran use case work either. > - If you build the Fortran code as a static library (rather common...), > then each pyx file will have their own copy. This will link successfully > but likely have a rather poor effect. So? lxml has two main modules, and if you build it statically against libxml2/libxslt, you end up with two static versions of that library in the two modules. Takes up some additional space, but otherwise works beautifully. Stefan From dagss at student.matnat.uio.no Wed Mar 2 11:54:44 2011 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Mar 2011 11:54:44 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E2079.9000102@behnel.de> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> Message-ID: <4D6E21F4.5030006@student.matnat.uio.no> On 03/02/2011 11:48 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 02.03.2011 11:20: >> c) Somehow provide more than one module in the same compilation unit. >> Again, this requires the build to work correctly, but seems less >> dangerous, >> and also has the advantage of *allowing* static linking of the Fortran >> library, if one wants to. >> >> But is something like this possible? Could one implement >> >> cython -o assembly.c file1.pyx file2.pyx file3.pyx >> >> ...where assembly.so would contain three Python modules? (initfile1, >> initfile2, and so on...) > > Can't currently work because the three modules would define the same > static C names. This could be fixed as part of the PEP 3121 > implementation: > > http://trac.cython.org/cython_trac/ticket/173 > > Or it could be worked around by overriding the prefixes in Naming.py > (which sounds ugly). > > Generally speaking, I'm -1 on this idea. I don't see a real use case, > and you're saying yourself that this isn't required to make your > Fortran use case work either. But assuming work is spent on Cython, it *could* work? I.e. there's not a problem with import mechanisms; Python assumes one module per .so or similar? Or would one have to write a custom importer? > > >> - If you build the Fortran code as a static library (rather common...), >> then each pyx file will have their own copy. This will link successfully >> but likely have a rather poor effect. > > So? lxml has two main modules, and if you build it statically against > libxml2/libxslt, you end up with two static versions of that library > in the two modules. Takes up some additional space, but otherwise > works beautifully. Problem is that Fortran code often has...interesting...programming practices. Global variables abound, and are often initialised between modules. Imagine: settings_mod.set_alpha(0.34) print compute_mod.get_alpha_squared() This behaves quite differently with two static versions rather than one... Dag Sverre From stefan_ml at behnel.de Wed Mar 2 12:35:05 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 02 Mar 2011 12:35:05 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E21F4.5030006@student.matnat.uio.no> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> Message-ID: <4D6E2B69.7040404@behnel.de> Dag Sverre Seljebotn, 02.03.2011 11:54: > On 03/02/2011 11:48 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 02.03.2011 11:20: >>> c) Somehow provide more than one module in the same compilation unit. >>> Again, this requires the build to work correctly, but seems less dangerous, >>> and also has the advantage of *allowing* static linking of the Fortran >>> library, if one wants to. >>> >>> But is something like this possible? Could one implement >>> >>> cython -o assembly.c file1.pyx file2.pyx file3.pyx >>> >>> ...where assembly.so would contain three Python modules? (initfile1, >>> initfile2, and so on...) >> >> Can't currently work because the three modules would define the same >> static C names. This could be fixed as part of the PEP 3121 implementation: >> >> http://trac.cython.org/cython_trac/ticket/173 >> >> Or it could be worked around by overriding the prefixes in Naming.py >> (which sounds ugly). >> >> Generally speaking, I'm -1 on this idea. I don't see a real use case, and >> you're saying yourself that this isn't required to make your Fortran use >> case work either. > > But assuming work is spent on Cython, it *could* work? I.e. there's not a > problem with import mechanisms You can have as many modules in a .so as you like, see the embedding support, for example. You just have to find a way to import them explicitly, which usually requires some call into the .so (as CPython does when calling the initmodule() function) to let it register the modules. Potentially, you could have one module in there that has the name of the .so file, and let that register the other modules automatically in its initmodule() function. Similar to a package's __init__.py that can import other modules from the package when it gets imported itself. You could think of your feature as a "package in a module". Actually, if Cython had direct support for this, it could simply join the static definitions in the separate modules into one, including merged string constants etc. >>> - If you build the Fortran code as a static library (rather common...), >>> then each pyx file will have their own copy. This will link successfully >>> but likely have a rather poor effect. >> >> So? lxml has two main modules, and if you build it statically against >> libxml2/libxslt, you end up with two static versions of that library in >> the two modules. Takes up some additional space, but otherwise works >> beautifully. > > Problem is that Fortran code often has...interesting...programming > practices. Global variables abound, and are often initialised between > modules. Imagine: > > settings_mod.set_alpha(0.34) > print compute_mod.get_alpha_squared() > > This behaves quite differently with two static versions rather than one... Then I'd suggest always linking dynamically. Stefan From dalcinl at gmail.com Wed Mar 2 16:11:14 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 2 Mar 2011 12:11:14 -0300 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E2B69.7040404@behnel.de> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> Message-ID: On 2 March 2011 08:35, Stefan Behnel wrote: > Dag Sverre Seljebotn, 02.03.2011 11:54: >> >> Problem is that Fortran code often has...interesting...programming >> practices. Global variables abound, and are often initialised between >> modules. Imagine: >> >> settings_mod.set_alpha(0.34) >> print compute_mod.get_alpha_squared() >> >> This behaves quite differently with two static versions rather than one... > > Then I'd suggest always linking dynamically. > And where are you going to put your fortran shared libraries? Dynamic linking details varies wildly across platforms... I'm very much understand Dag's use case and concerns, and I do think that some research in all this is worth it. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From d.s.seljebotn at astro.uio.no Wed Mar 2 16:37:16 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Mar 2011 16:37:16 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> Message-ID: <4D6E642C.9010001@student.matnat.uio.no> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: > On 2 March 2011 08:35, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 02.03.2011 11:54: >>> Problem is that Fortran code often has...interesting...programming >>> practices. Global variables abound, and are often initialised between >>> modules. Imagine: >>> >>> settings_mod.set_alpha(0.34) >>> print compute_mod.get_alpha_squared() >>> >>> This behaves quite differently with two static versions rather than one... >> Then I'd suggest always linking dynamically. >> > And where are you going to put your fortran shared libraries? Dynamic > linking details varies wildly across platforms... I'm very much > understand Dag's use case and concerns, and I do think that some > research in all this is worth it. I'm not sure if there's much more to research at the moment -- Stefan says it is possible, and that's what I wanted to know at this stage. If I want it, I obviously need to implement it myself. (And if such a patch implements PEP 3121 and there's a demonstrated need for it with some users, I really can't see it getting rejected just out of it being in "poor taste"). I.e., I'm going to make Fwrap spit out multiple pyx files and worry about this later. If multiple .pyx in one .so was fundamentally impossible, I might have gone another route with Fwrap. That was all. Thanks Stefan!, Dag Sverre From stefan_ml at behnel.de Wed Mar 2 17:01:41 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 02 Mar 2011 17:01:41 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E642C.9010001@student.matnat.uio.no> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> Message-ID: <4D6E69E5.8000806@behnel.de> Dag Sverre Seljebotn, 02.03.2011 16:37: > On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: >> On 2 March 2011 08:35, Stefan Behnel wrote: >>> Dag Sverre Seljebotn, 02.03.2011 11:54: >>>> Problem is that Fortran code often has...interesting...programming >>>> practices. Global variables abound, and are often initialised between >>>> modules. Imagine: >>>> >>>> settings_mod.set_alpha(0.34) >>>> print compute_mod.get_alpha_squared() >>>> >>>> This behaves quite differently with two static versions rather than one... >>> Then I'd suggest always linking dynamically. >>> >> And where are you going to put your fortran shared libraries? Dynamic >> linking details varies wildly across platforms... I'm very much >> understand Dag's use case and concerns, and I do think that some >> research in all this is worth it. > > I'm not sure if there's much more to research at the moment -- Stefan says > it is possible, and that's what I wanted to know at this stage. If I want > it, I obviously need to implement it myself. (And if such a patch > implements PEP 3121 and there's a demonstrated need for it with some users, > I really can't see it getting rejected just out of it being in "poor taste"). > > I.e., I'm going to make Fwrap spit out multiple pyx files and worry about > this later. If multiple .pyx in one .so was fundamentally impossible, I > might have gone another route with Fwrap. That was all. The feature I could imagine becoming part of Cython is "compiling packages". I.e. you'd call "cython" on a package and it would output a directory with a single __init__.so that contains the modules compiled from all .pyx/.py files in that package. Importing the package would then trigger an import of that __init__.so, which in turn will execute code in its init__init__() function to register the other modules. Would that work for you? Stefan From d.s.seljebotn at astro.uio.no Wed Mar 2 18:31:49 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 02 Mar 2011 18:31:49 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E69E5.8000806@behnel.de> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> Message-ID: <4D6E7F05.6080108@student.matnat.uio.no> On 03/02/2011 05:01 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 02.03.2011 16:37: >> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: >>> On 2 March 2011 08:35, Stefan Behnel wrote: >>>> Dag Sverre Seljebotn, 02.03.2011 11:54: >>>>> Problem is that Fortran code often has...interesting...programming >>>>> practices. Global variables abound, and are often initialised between >>>>> modules. Imagine: >>>>> >>>>> settings_mod.set_alpha(0.34) >>>>> print compute_mod.get_alpha_squared() >>>>> >>>>> This behaves quite differently with two static versions rather >>>>> than one... >>>> Then I'd suggest always linking dynamically. >>>> >>> And where are you going to put your fortran shared libraries? Dynamic >>> linking details varies wildly across platforms... I'm very much >>> understand Dag's use case and concerns, and I do think that some >>> research in all this is worth it. >> >> I'm not sure if there's much more to research at the moment -- Stefan >> says >> it is possible, and that's what I wanted to know at this stage. If I >> want >> it, I obviously need to implement it myself. (And if such a patch >> implements PEP 3121 and there's a demonstrated need for it with some >> users, >> I really can't see it getting rejected just out of it being in "poor >> taste"). >> >> I.e., I'm going to make Fwrap spit out multiple pyx files and worry >> about >> this later. If multiple .pyx in one .so was fundamentally impossible, I >> might have gone another route with Fwrap. That was all. > > The feature I could imagine becoming part of Cython is "compiling > packages". I.e. you'd call "cython" on a package and it would output a > directory with a single __init__.so that contains the modules compiled > from all .pyx/.py files in that package. Importing the package would > then trigger an import of that __init__.so, which in turn will execute > code in its init__init__() function to register the other modules. > > Would that work for you? Yes, that sounds like exactly what I'm after. Dag Sverre From dalcinl at gmail.com Wed Mar 2 19:58:43 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 2 Mar 2011 15:58:43 -0300 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E69E5.8000806@behnel.de> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> Message-ID: On 2 March 2011 13:01, Stefan Behnel wrote: > Dag Sverre Seljebotn, 02.03.2011 16:37: >> >> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote: >>> >>> On 2 March 2011 08:35, Stefan Behnel wrote: >>>> >>>> Dag Sverre Seljebotn, 02.03.2011 11:54: >>>>> >>>>> Problem is that Fortran code often has...interesting...programming >>>>> practices. Global variables abound, and are often initialised between >>>>> modules. Imagine: >>>>> >>>>> settings_mod.set_alpha(0.34) >>>>> print compute_mod.get_alpha_squared() >>>>> >>>>> This behaves quite differently with two static versions rather than >>>>> one... >>>> >>>> Then I'd suggest always linking dynamically. >>>> >>> And where are you going to put your fortran shared libraries? Dynamic >>> linking details varies wildly across platforms... I'm very much >>> understand Dag's use case and concerns, and I do think that some >>> research in all this is worth it. >> >> I'm not sure if there's much more to research at the moment -- Stefan says >> it is possible, and that's what I wanted to know at this stage. If I want >> it, I obviously need to implement it myself. (And if such a patch >> implements PEP 3121 and there's a demonstrated need for it with some >> users, >> I really can't see it getting rejected just out of it being in "poor >> taste"). >> >> I.e., I'm going to make Fwrap spit out multiple pyx files and worry about >> this later. If multiple .pyx in one .so was fundamentally impossible, I >> might have gone another route with Fwrap. That was all. > > The feature I could imagine becoming part of Cython is "compiling packages". > I.e. you'd call "cython" on a package and it would output a directory with a > single __init__.so that contains the modules compiled from all .pyx/.py > files in that package. Importing the package would then trigger an import of > that __init__.so, which in turn will execute code in its init__init__() > function to register the other modules. > $ cd /tmp/ $ mkdir pkg $ touch pkg/__init__.so $ python -c 'import pkg' Traceback (most recent call last): File "", line 1, in ImportError: No module named pkg So that's not going to work... The import machinery ignores __init__.so files. Instead, you need a "package.so" file, the initpackage() init function should in turn setup the package structure and initialize submodules. I've sent privately to Dag an example of how this should be done. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From vitja.makarov at gmail.com Wed Mar 2 20:18:47 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 2 Mar 2011 22:18:47 +0300 Subject: [Cython] "__ipow__ has wrong number of arguments" is back here Message-ID: Hi! I noticed that this error came back again. https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ -- vitja. From dalcinl at gmail.com Wed Mar 2 21:14:29 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 2 Mar 2011 17:14:29 -0300 Subject: [Cython] "__ipow__ has wrong number of arguments" is back here In-Reply-To: References: Message-ID: On 2 March 2011 16:18, Vitja Makarov wrote: > Hi! > > I noticed that this error came back again. > > https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ > I've pushed a fix for __ipow__ to take just two arguments, like regular Python classes. https://github.com/cython/cython/commit/490fc80e6b494f8ed65f78fc2032d856ec75b8ef But I forgot to update test for ticket #561, now this should fixed: https://github.com/cython/cython/commit/829d6485b77b5688671959f840712c01e6886556 -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From vitja.makarov at gmail.com Wed Mar 2 22:27:20 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 3 Mar 2011 00:27:20 +0300 Subject: [Cython] strange bug in starred target In-Reply-To: References: Message-ID: 2011/3/3 Vitja Makarov : > This doesn't work: > def assign(): > ? ?a, *b = 1,2,3,4,5 > ? ?return a, b > >>>> import x >>>> x.assign() > Traceback (most recent call last): > ?File "", line 1, in > ?File "x.pyx", line 6, in x.assign (x.c:445) > ? ?a, *b = 1,2,3,4,5 > TypeError: Expected list, got int > > And this is ok: > > def assign(): > ? ?a, *b = 1,2,3,4,5 > ? ?return a, b > ? ?*a, b = 1,2,3,4,5 > ? ?return a, b > >>>> import x >>>> x.assign() > (1, [2, 3, 4, 5]) > > I hope problem is somewhere in map_starred_assignment() -- vitja. From greg.ewing at canterbury.ac.nz Thu Mar 3 01:01:20 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 03 Mar 2011 13:01:20 +1300 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6E69E5.8000806@behnel.de> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> Message-ID: <4D6EDA50.5030807@canterbury.ac.nz> Stefan Behnel wrote: > you'd call "cython" on a package and it would output a > directory with a single __init__.so that contains the modules compiled > from all .pyx/.py files in that package. Importing the package would > then trigger an import of that __init__.so, which in turn will execute > code in its init__init__() function to register the other modules. I don't think it even has to be a directory with an __init__, it could just be an ordinary .so file with the name of the package. I just tried an experiment in Python: # onefilepackage.py import new, sys blarg = new.module("blarg") blarg.thing = "This is the thing" sys.modules["onefilepackage.blarg"] = blarg and two different ways of importing it: >>> from onefilepackage import blarg >>> blarg >>> blarg.thing 'This is the thing' >>> import onefilepackage.blarg >>> onefilepackage.blarg.thing 'This is the thing' So assuming the same thing works with a .so instead of a .py, all you need to do is emit a .so whose init function stuffs appropriate entries into sys.modules to make it look like a package. -- Greg From wking at drexel.edu Thu Mar 3 01:51:04 2011 From: wking at drexel.edu (W. Trevor King) Date: Wed, 2 Mar 2011 19:51:04 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <20110219233126.GC25132@tyr.home.net> <20110222175542.GA8312@tyr.home.net> <4D640BFD.4050708@behnel.de> <20110222200207.GA14712@tyr.home.net> <20110226114826.GA12380@tyr.home.net> <20110227021409.GA3698@tyr.home.net> Message-ID: <20110303005103.GA12710@tyr.home.net> On Sat, Feb 26, 2011 at 07:11:48PM -0800, Robert Bradshaw wrote: > On Sat, Feb 26, 2011 at 6:14 PM, W. Trevor King wrote: > > On Sat, Feb 26, 2011 at 10:01:43AM -0800, Robert Bradshaw wrote: > >> On Sat, Feb 26, 2011 at 3:48 AM, W. Trevor King wrote: > >> > On Fri, Feb 25, 2011 at 11:11:03PM -0800, Robert Bradshaw wrote: > >> >> On Tue, Feb 22, 2011 at 12:02 PM, W. Trevor King wrote: > >> >> > An easy, if uglier, workaround would be to prepend attributes with the > >> >> > class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity. > >> >> > Then the Ctx class could subclass the current CtxAttribute classes > >> >> > instead of binding instances of each of them. That way Ctx would keep > >> >> > its traditional flat attribute namespace and easy deepcopy, but > >> >> > eveyone in Nodes, etc. that will use the attributes would become > >> >> > class-name dependent. > >> >> > >> >> I'd be up for flattening this. In particular, changing every > >> >> "entry.name" to "entry.python_binding.name" seems to be a lot of churn > >> >> and extra verbiage for not much benefit. The only overlap I see is > >> >> name and visibility, and keeping name/cname and adding cvisibility > >> >> would be preferable to me. > >> > > >> > That works for me, but I see possible ambiguity in cname. From the > >> > "external C code" docs: > >> > > >> > The other way is to use a C name specification to give different > >> > Cython and C names to the C function. Suppose, for example, that > >> > you want to wrap an external function called eject_tomato(). If > >> > you declare it as: > >> > > >> > cdef extern void c_eject_tomato "eject_tomato" (float speed) > >> > > >> > then its name inside the Cython module will be c_eject_tomato, > >> > whereas its name in C will be eject_tomato. > >> > > >> > In this case I was eventually going to use > >> > > >> > c_source.name = eject_tomato > >> > c_source.namespace = ... > >> > c_binding.name = c_eject_tomato > >> > c_binding.namespace = ... > >> > python_binding.name = eject_tomato > >> > > >> > I'm not sure how Cython handles name/cname with this case at the > >> > moment, but it seems like there are two cnames to keep track of. > >> > >> In this case, cname is "eject_tomato" (which actually gets emitted in > >> the C source file to use it) and name (in both Python and Cython > >> namespace) is "c_eject_tomato." > > > > What is the "Cython namespace"? Is that what you get when you cimport > > something (vs. the Python namespace being what you get when you import > > something). > > Yes, that would be a good way to understand it. Okay, I'm back to a flat attribute namespace in my branch, and I'm moving through Nodes removing the places where I currently collapse back to an overloaded visibility. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From dalcinl at gmail.com Thu Mar 3 02:47:45 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 2 Mar 2011 22:47:45 -0300 Subject: [Cython] "__ipow__ has wrong number of arguments" is back here In-Reply-To: References: Message-ID: On 2 March 2011 17:14, Lisandro Dalcin wrote: > On 2 March 2011 16:18, Vitja Makarov wrote: >> Hi! >> >> I noticed that this error came back again. >> >> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ >> > > I've pushed a fix for __ipow__ to take just two arguments, like > regular Python classes. > https://github.com/cython/cython/commit/490fc80e6b494f8ed65f78fc2032d856ec75b8ef > > But I forgot to update test for ticket #561, now this should fixed: > https://github.com/cython/cython/commit/829d6485b77b5688671959f840712c01e6886556 > BTW, do you all agree with this change? I expect this change to break some code, particularly in Sage... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From wking at drexel.edu Thu Mar 3 02:54:47 2011 From: wking at drexel.edu (W. Trevor King) Date: Wed, 2 Mar 2011 20:54:47 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <4D60DE3C.3070306@behnel.de> <4D618506.6070702@behnel.de> Message-ID: <20110303015446.GA27504@tyr.home.net> Previous discussions in this thread [1,2] have discussed the issues associated with overloading the 'public' keyword. For an example of the difficulties this causes, see the ugly workaround [3] in my recent commit [4]. Definately worth fixing this syntax. How do syntax changes with deprecations work? The Parsing module doesn't seem to be setup to handle things like that. [1] http://mail.python.org/pipermail/cython-devel/2011-February/000081.html [2] http://mail.python.org/pipermail/cython-devel/2011-February/000086.html [3] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f#L1R2552 [4] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Thu Mar 3 03:08:12 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 2 Mar 2011 18:08:12 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110303015446.GA27504@tyr.home.net> References: <4D60DE3C.3070306@behnel.de> <4D618506.6070702@behnel.de> <20110303015446.GA27504@tyr.home.net> Message-ID: On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King wrote: > Previous discussions in this thread [1,2] have discussed the issues > associated with overloading the 'public' keyword. ?For an example of > the difficulties this causes, see the ugly workaround [3] in my recent > commit [4]. ?Definately worth fixing this syntax. > > How do syntax changes with deprecations work? ?The Parsing module > doesn't seem to be setup to handle things like that. It's not. We rarely deprecate syntax, and we should at least give people a chance to move away from the old syntax first to the new first, so we'll have to support both for a bit at least. When we do deprecate the old way of doing things, I think we'll just use Errors.warning(...). In the long run it'd be nice to have a formal grammar for Cython and use a generated parser rather than maintaining this all by hand, but Parsing.py does more than just parsing at the moment. - Robert > [1] http://mail.python.org/pipermail/cython-devel/2011-February/000081.html > [2] http://mail.python.org/pipermail/cython-devel/2011-February/000086.html > [3] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f#L1R2552 > [4] https://github.com/wking/cython/commit/a918d1466928ea712817df18c3bc66d9c4c5c53f > > -- > This email may be signed or encrypted with GPG (http://www.gnupg.org). > The GPG signature (if present) will be attached as 'signature.asc'. > For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy > > My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > From dalcinl at gmail.com Thu Mar 3 05:38:09 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 3 Mar 2011 01:38:09 -0300 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6EDA50.5030807@canterbury.ac.nz> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> Message-ID: On 2 March 2011 21:01, Greg Ewing wrote: > Stefan Behnel wrote: >> >> you'd call "cython" on a package and it would output a directory with a >> single __init__.so that contains the modules compiled from all .pyx/.py >> files in that package. Importing the package would then trigger an import of >> that __init__.so, which in turn will execute code in its init__init__() >> function to register the other modules. > > I don't think it even has to be a directory with an __init__, > it could just be an ordinary .so file with the name of the > package. > > I just tried an experiment in Python: > > # onefilepackage.py > import new, sys > blarg = new.module("blarg") > blarg.thing = "This is the thing" > sys.modules["onefilepackage.blarg"] = blarg > > and two different ways of importing it: > >>>> from onefilepackage import blarg >>>> blarg > >>>> blarg.thing > 'This is the thing' > >>>> import onefilepackage.blarg >>>> onefilepackage.blarg.thing > 'This is the thing' > I'm hacking around these lines. However, I'm working to maintain different modules in different C compilation units, in order to workaround the obvious issue with duplicated global C symbols. > So assuming the same thing works with a .so instead of a .py, > all you need to do is emit a .so whose init function stuffs > appropriate entries into sys.modules to make it look like > a package. > However, the import machinery does not treat .so the same as *.pyx. For example, I have a problem with Python 3. For .py modules, before the module code starts to execute, the matching entries in sys.modules is already there. The same happens in Python 2 for .so modules, as Py_InitModule() add the entry in sys.modules early. However, in Python 3 that is not te case (and only for the .so, for .py is the same as in Py2), the import machinery adds the entry later, after the finalization of the module init function. I'm tempted to workaround this by setting the entry in sys.modules right after the call to PyModule_Create() ... What do you think about this? Any potential issue? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Thu Mar 3 07:43:19 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 03 Mar 2011 07:43:19 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> Message-ID: <4D6F3887.3090906@behnel.de> Lisandro Dalcin, 03.03.2011 05:38: > On 2 March 2011 21:01, Greg Ewing wrote: >> Stefan Behnel wrote: >>> >>> you'd call "cython" on a package and it would output a directory with a >>> single __init__.so that contains the modules compiled from all .pyx/.py >>> files in that package. Importing the package would then trigger an import of >>> that __init__.so, which in turn will execute code in its init__init__() >>> function to register the other modules. >> >> I don't think it even has to be a directory with an __init__, >> it could just be an ordinary .so file with the name of the >> package. >> >> I just tried an experiment in Python: >> >> # onefilepackage.py >> import new, sys >> blarg = new.module("blarg") >> blarg.thing = "This is the thing" >> sys.modules["onefilepackage.blarg"] = blarg >> >> and two different ways of importing it: >> >> >>> from onefilepackage import blarg >> >>> blarg >> >> >>> blarg.thing >> 'This is the thing' >> >> >>> import onefilepackage.blarg >> >>> onefilepackage.blarg.thing >> 'This is the thing' >> > > I'm hacking around these lines. However, I'm working to maintain > different modules in different C compilation units, in order to > workaround the obvious issue with duplicated global C symbols. That should be ok as a first attempt to get it working quickly. I'd still like to see the modules merged in the long term in order to increase the benefit of the more compact format. They'd all share the same code generator and Cython's C internals, C helper functions, constants, builtins, etc., but each of them would use a separate (name mangling) scope to keep the visible C names separate. >> So assuming the same thing works with a .so instead of a .py, >> all you need to do is emit a .so whose init function stuffs >> appropriate entries into sys.modules to make it look like >> a package. >> > > However, the import machinery does not treat .so the same as *.pyx. > For example, I have a problem with Python 3. For .py modules, before > the module code starts to execute, the matching entries in sys.modules > is already there. And it has to be, in order to prevent accidental reimports. > The same happens in Python 2 for .so modules, as > Py_InitModule() add the entry in sys.modules early. However, in Python > 3 that is not te case (and only for the .so, for .py is the same as in > Py2), the import machinery adds the entry later, after the > finalization of the module init function. I'm tempted to workaround > this by setting the entry in sys.modules right after the call to > PyModule_Create() ... What do you think about this? Any potential > issue? No idea. I'd say, read the source and give it a try. Stefan From vitja.makarov at gmail.com Thu Mar 3 08:38:17 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 3 Mar 2011 10:38:17 +0300 Subject: [Cython] Control flow graph In-Reply-To: References: <4D5A2D6E.9060406@behnel.de> <4D641C26.9070705@behnel.de> Message-ID: 2011/2/28 Vitja Makarov : > 2011/2/23 Vitja Makarov : >> 2011/2/23 Vitja Makarov : >>> 2011/2/22 Stefan Behnel : >>>> Vitja Makarov, 20.02.2011 18:23: >>>>> >>>>> 2011/2/16 Vitja Makarov: >>>>>> >>>>>> Hmm... both python and codespeaks in the thread >>>> >>>> Yes, we should keep it to cython-devel only. Sorry for mixing it up. >>>> >>>> >>>>>> Here is my commit it's mostly broken now but anyway >>>>>> >>>>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a >>>> >>>> Flow control support is large enough to merit its own module. Not sure how >>>> 'smart' git is here, but you can always keep the history by explicitly >>>> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated >>>> sections from both files. >>>> >>> >>> Ok. >>> >> >> Oops there is no copy command in git. >> >> >>>> You are duplicating some code from the type inferencer. We might want to >>>> clean that up at some point. However, given that flow control analysis will >>>> allow us to improve the type inferencer, I think it's best to keep this code >>>> in the FCA part. >>>> >>> >>> Yes, I think it could replace MarkAssignments transform later. >>> Unreachable code could be delete there too. >>> >>>> >>>>> I've update stuff: >>>>> ?- algo for finding definitions >>>>> ?- warnings for uninitialized and may be uninitialised use >>>>> ?- few test cases >>>> >>>> That looks very nice so far. Any idea how well it scales? >>>> >>> >>> "Usually iterative algorithm takes no more then 5 iterations" >>> >>> For ExprNodes.py max number is 15 while avg is about 3 >>> >>> About execution time: >>> >>> ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325, >>> ~10% slow down. >>> -O flag could be introduced but I don't think that's a good idea. >>> >>> Should later try to execute cython compiled code. >>> >>>> >>>>> Trying to compile ParseTreeTransforms.py I've found this for example: >>>>> >>>>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable >>>>> 'template' may be used uninitialized >>>>> >>>>> ? ? def create_Property(self, entry): >>>>> ? ? ? ? if entry.visibility == 'public': >>>>> ? ? ? ? ? ? if entry.type.is_pyobject: >>>>> ? ? ? ? ? ? ? ? template = self.basic_pyobject_property >>>>> ? ? ? ? ? ? else: >>>>> ? ? ? ? ? ? ? ? template = self.basic_property >>>>> ? ? ? ? elif entry.visibility == 'readonly': >>>>> ? ? ? ? ? ? template = self.basic_property_ro >>>>> ? ? ? ? property = template.substitute({ >>>>> ? ? ? ? ? ? ? ? u"ATTR": ExprNodes.AttributeNode(pos=entry.pos, >>>>> >>>>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"), >>>>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?attribute=entry.name), >>>>> ? ? ? ? ? ? }, pos=entry.pos).stats[0] >>>> >>>> Ok, I guess that code generally works, but it's better to get rid of the >>>> code smell. >>>> >>> >>> Might be used warning should be disabled by default, because algorithm >>> isn't smart enough: >>> >>> a = 1 >>> if (a): b = 1 >>> if (a): print b >>> >>> See also: >>> http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325 >>> > > > I've updated things: > > ?- Now it passes all the tests (some tests are run with > remove_unreachable=False) > ?- Control flow graph support all the control statements (hope so) > ?- Analyse PyClassDefNode (now only inner, need some more work) > ?- Now we have the following optional warnings: > ?1. uninitialized variable use [default on] > ?2. possibly uninitialized use [-Wextra] > ?3. unreachable code [default on] (this needs rework, unreachable > code detection could be better handled inside CreateControlFlowGraph) > ?4. unused variable [-Wextra] > ?5. unused argument [-X warn.unused_arg=True] > ?6. unused result [-X warn_unused_result=True] > > ?- Optional dot graph output: > > ?$ cython foo.pyx -X control_flow.dot_output=foo.dot > ?$ dot -Tpng -o foo.png foo.dot > I've pushed unused varaiable/result removal. Unused argument removal require some more work, especially generic args and kwargs. Now I'm going to handle local variable deletion and then remove Py_None initialization. -- vitja. From markflorisson88 at gmail.com Thu Mar 3 10:32:15 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 3 Mar 2011 10:32:15 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6F3887.3090906@behnel.de> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> <4D6F3887.3090906@behnel.de> Message-ID: On 3 March 2011 07:43, Stefan Behnel wrote: > Lisandro Dalcin, 03.03.2011 05:38: >> >> On 2 March 2011 21:01, Greg Ewing ?wrote: >>> >>> Stefan Behnel wrote: >>>> >>>> you'd call "cython" on a package and it would output a directory with a >>>> single __init__.so that contains the modules compiled from all .pyx/.py >>>> files in that package. Importing the package would then trigger an >>>> import of >>>> that __init__.so, which in turn will execute code in its init__init__() >>>> function to register the other modules. >>> >>> I don't think it even has to be a directory with an __init__, >>> it could just be an ordinary .so file with the name of the >>> package. >>> >>> I just tried an experiment in Python: >>> >>> # onefilepackage.py >>> import new, sys >>> blarg = new.module("blarg") >>> blarg.thing = "This is the thing" >>> sys.modules["onefilepackage.blarg"] = blarg >>> >>> and two different ways of importing it: >>> >>> >>> from onefilepackage import blarg >>> >>> blarg >>> >>> >>> blarg.thing >>> 'This is the thing' >>> >>> >>> import onefilepackage.blarg >>> >>> onefilepackage.blarg.thing >>> 'This is the thing' >>> >> >> I'm hacking around these lines. However, I'm working to maintain >> different modules in different C compilation units, in order to >> workaround the obvious issue with duplicated global C symbols. > > That should be ok as a first attempt to get it working quickly. I'd still > like to see the modules merged in the long term in order to increase the > benefit of the more compact format. They'd all share the same code generator > and Cython's C internals, C helper functions, constants, builtins, etc., but > each of them would use a separate (name mangling) scope to keep the visible > C names separate. I was thinking that perhaps we could share declarations common to all cython modules (compiled with that version of Cython) in a cython.h header file (which is also imho nicer to maintain than a code.put() in e.g. ModuleNode.generate_module_preamble), and put it in e.g. Cython/Includes and set the -I C compiler flag to point to the Includes directory. Module-specific functions would still be declared static, of course. And if users want to ship generated C files to avoid Cython as a dependency, they could simply ship the header and adjust their setup.py. If you want to merge modules and the "package-approach" is chosen, it should have well-defined semantics for in-place builds, and package/__init__.py is preferred over package.so. So how would you solve that problem without either placing package.so in the package itself, or giving it another name (and perhaps star-importing it from __init__.py)? Basically, if people want to combine several modules into one they could use the 'include' statement. e.g. in spam.pyx you 'include "ham.pyx"' and in spam.pxd you 'include "ham.pxd"'. (although you'd probably rename ham.pyx to ham.pxi, and you'd probably merge spam.pxd with ham.pxd) In any case, I'm just wondering, would this functionality be more useful than our current include statement and a cython.h header that is shared by default? >>> So assuming the same thing works with a .so instead of a .py, >>> all you need to do is emit a .so whose init function stuffs >>> appropriate entries into sys.modules to make it look like >>> a package. >>> >> >> However, the import machinery does not treat .so the same as *.pyx. >> For example, I have a problem with Python 3. For .py modules, before >> the module code starts to execute, the matching entries in sys.modules >> is already there. > > And it has to be, in order to prevent accidental reimports. > > >> The same happens in Python 2 for .so modules, as >> Py_InitModule() add the entry in sys.modules early. However, in Python >> 3 that is not te case (and only for the .so, for .py is the same as in >> Py2), the import machinery adds the entry later, after the >> finalization of the module init function. I'm tempted to workaround >> this by setting the entry in sys.modules right after the call to >> PyModule_Create() ... What do you think about this? Any potential >> issue? > > No idea. I'd say, read the source and give it a try. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From vitja.makarov at gmail.com Thu Mar 3 10:48:19 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 3 Mar 2011 12:48:19 +0300 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> <4D6F3887.3090906@behnel.de> Message-ID: 2011/3/3 mark florisson : > On 3 March 2011 07:43, Stefan Behnel wrote: >> Lisandro Dalcin, 03.03.2011 05:38: >>> >>> On 2 March 2011 21:01, Greg Ewing ?wrote: >>>> >>>> Stefan Behnel wrote: >>>>> >>>>> you'd call "cython" on a package and it would output a directory with a >>>>> single __init__.so that contains the modules compiled from all .pyx/.py >>>>> files in that package. Importing the package would then trigger an >>>>> import of >>>>> that __init__.so, which in turn will execute code in its init__init__() >>>>> function to register the other modules. >>>> >>>> I don't think it even has to be a directory with an __init__, >>>> it could just be an ordinary .so file with the name of the >>>> package. >>>> >>>> I just tried an experiment in Python: >>>> >>>> # onefilepackage.py >>>> import new, sys >>>> blarg = new.module("blarg") >>>> blarg.thing = "This is the thing" >>>> sys.modules["onefilepackage.blarg"] = blarg >>>> >>>> and two different ways of importing it: >>>> >>>> >>> from onefilepackage import blarg >>>> >>> blarg >>>> >>>> >>> blarg.thing >>>> 'This is the thing' >>>> >>>> >>> import onefilepackage.blarg >>>> >>> onefilepackage.blarg.thing >>>> 'This is the thing' >>>> >>> >>> I'm hacking around these lines. However, I'm working to maintain >>> different modules in different C compilation units, in order to >>> workaround the obvious issue with duplicated global C symbols. >> >> That should be ok as a first attempt to get it working quickly. I'd still >> like to see the modules merged in the long term in order to increase the >> benefit of the more compact format. They'd all share the same code generator >> and Cython's C internals, C helper functions, constants, builtins, etc., but >> each of them would use a separate (name mangling) scope to keep the visible >> C names separate. > > I was thinking that perhaps we could share declarations common to all > cython modules (compiled with that version of Cython) in a cython.h > header file (which is also imho nicer to maintain than a code.put() in > e.g. ModuleNode.generate_module_preamble), and put it in e.g. > Cython/Includes and set the -I C compiler flag to point to the > Includes directory. > Module-specific functions would still be declared static, of course. > And if users want to ship generated C files to avoid Cython as a > dependency, they could simply ship the header and adjust their > setup.py. > > If you want to merge modules and the "package-approach" is chosen, it > should have well-defined semantics for in-place builds, and > package/__init__.py is preferred over package.so. So how would you > solve that problem without either placing package.so in the package > itself, or giving it another name (and perhaps star-importing it from > __init__.py)? Basically, if people want to combine several modules > into one they could use the 'include' statement. > > e.g. in spam.pyx you 'include "ham.pyx"' and in spam.pxd you 'include > "ham.pxd"'. > > (although you'd probably rename ham.pyx to ham.pxi, and you'd probably > merge spam.pxd with ham.pxd) > > In any case, I'm just wondering, would this functionality be more > useful than our current include statement and a cython.h header that > is shared by default? > >>>> So assuming the same thing works with a .so instead of a .py, >>>> all you need to do is emit a .so whose init function stuffs >>>> appropriate entries into sys.modules to make it look like >>>> a package. >>>> >>> >>> However, the import machinery does not treat .so the same as *.pyx. >>> For example, I have a problem with Python 3. For .py modules, before >>> the module code starts to execute, the matching entries in sys.modules >>> is already there. >> >> And it has to be, in order to prevent accidental reimports. >> >> >>> The same happens in Python 2 for .so modules, as >>> Py_InitModule() add the entry in sys.modules early. However, in Python >>> 3 that is not te case (and only for the .so, for .py is the same as in >>> Py2), the import machinery adds the entry later, after the >>> finalization of the module init function. I'm tempted to workaround >>> this by setting the entry in sys.modules right after the call to >>> PyModule_Create() ... What do you think about this? Any potential >>> issue? >> >> No idea. I'd say, read the source and give it a try. >> >> Stefan To share common sources is a good idea, we can also share "code" in libcython-.so But then we should handle ABI compatibility problems. -- vitja. From markflorisson88 at gmail.com Thu Mar 3 10:57:05 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 3 Mar 2011 10:57:05 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> <4D6F3887.3090906@behnel.de> Message-ID: On 3 March 2011 10:48, Vitja Makarov wrote: > 2011/3/3 mark florisson : >> On 3 March 2011 07:43, Stefan Behnel wrote: >>> Lisandro Dalcin, 03.03.2011 05:38: >>>> >>>> On 2 March 2011 21:01, Greg Ewing ?wrote: >>>>> >>>>> Stefan Behnel wrote: >>>>>> >>>>>> you'd call "cython" on a package and it would output a directory with a >>>>>> single __init__.so that contains the modules compiled from all .pyx/.py >>>>>> files in that package. Importing the package would then trigger an >>>>>> import of >>>>>> that __init__.so, which in turn will execute code in its init__init__() >>>>>> function to register the other modules. >>>>> >>>>> I don't think it even has to be a directory with an __init__, >>>>> it could just be an ordinary .so file with the name of the >>>>> package. >>>>> >>>>> I just tried an experiment in Python: >>>>> >>>>> # onefilepackage.py >>>>> import new, sys >>>>> blarg = new.module("blarg") >>>>> blarg.thing = "This is the thing" >>>>> sys.modules["onefilepackage.blarg"] = blarg >>>>> >>>>> and two different ways of importing it: >>>>> >>>>> >>> from onefilepackage import blarg >>>>> >>> blarg >>>>> >>>>> >>> blarg.thing >>>>> 'This is the thing' >>>>> >>>>> >>> import onefilepackage.blarg >>>>> >>> onefilepackage.blarg.thing >>>>> 'This is the thing' >>>>> >>>> >>>> I'm hacking around these lines. However, I'm working to maintain >>>> different modules in different C compilation units, in order to >>>> workaround the obvious issue with duplicated global C symbols. >>> >>> That should be ok as a first attempt to get it working quickly. I'd still >>> like to see the modules merged in the long term in order to increase the >>> benefit of the more compact format. They'd all share the same code generator >>> and Cython's C internals, C helper functions, constants, builtins, etc., but >>> each of them would use a separate (name mangling) scope to keep the visible >>> C names separate. >> >> I was thinking that perhaps we could share declarations common to all >> cython modules (compiled with that version of Cython) in a cython.h >> header file (which is also imho nicer to maintain than a code.put() in >> e.g. ModuleNode.generate_module_preamble), and put it in e.g. >> Cython/Includes and set the -I C compiler flag to point to the >> Includes directory. >> Module-specific functions would still be declared static, of course. >> And if users want to ship generated C files to avoid Cython as a >> dependency, they could simply ship the header and adjust their >> setup.py. >> >> If you want to merge modules and the "package-approach" is chosen, it >> should have well-defined semantics for in-place builds, and >> package/__init__.py is preferred over package.so. So how would you >> solve that problem without either placing package.so in the package >> itself, or giving it another name (and perhaps star-importing it from >> __init__.py)? Basically, if people want to combine several modules >> into one they could use the 'include' statement. >> >> e.g. in spam.pyx you 'include "ham.pyx"' and in spam.pxd you 'include >> "ham.pxd"'. >> >> (although you'd probably rename ham.pyx to ham.pxi, and you'd probably >> merge spam.pxd with ham.pxd) >> >> In any case, I'm just wondering, would this functionality be more >> useful than our current include statement and a cython.h header that >> is shared by default? >> >>>>> So assuming the same thing works with a .so instead of a .py, >>>>> all you need to do is emit a .so whose init function stuffs >>>>> appropriate entries into sys.modules to make it look like >>>>> a package. >>>>> >>>> >>>> However, the import machinery does not treat .so the same as *.pyx. >>>> For example, I have a problem with Python 3. For .py modules, before >>>> the module code starts to execute, the matching entries in sys.modules >>>> is already there. >>> >>> And it has to be, in order to prevent accidental reimports. >>> >>> >>>> The same happens in Python 2 for .so modules, as >>>> Py_InitModule() add the entry in sys.modules early. However, in Python >>>> 3 that is not te case (and only for the .so, for .py is the same as in >>>> Py2), the import machinery adds the entry later, after the >>>> finalization of the module init function. I'm tempted to workaround >>>> this by setting the entry in sys.modules right after the call to >>>> PyModule_Create() ... What do you think about this? Any potential >>>> issue? >>> >>> No idea. I'd say, read the source and give it a try. >>> >>> Stefan > > > To share common sources is a good idea, we can also share "code" in > libcython-.so > But then we should handle ABI compatibility problems. > > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > The idea sounds nice, but it would mean that Cython needs not only to be available, it should also be installed. If it's installed locally (and not in the system as root), then people would need to set CFLAGS when compiling and LD_LIBRARY_PATH when running if they wanted to use this functionality. But I guess if this feature is optional it could work great for systems with package managers (especially if libcython can be installed separately without Cython), and people who wish to distribute their project without the Cython dependency could avoid this option. From stefan_ml at behnel.de Thu Mar 3 10:58:14 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 03 Mar 2011 10:58:14 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> <4D6F3887.3090906@behnel.de> Message-ID: <4D6F6636.5030509@behnel.de> mark florisson, 03.03.2011 10:32: > On 3 March 2011 07:43, Stefan Behnel wrote: >> Lisandro Dalcin, 03.03.2011 05:38: >>> >>> On 2 March 2011 21:01, Greg Ewing wrote: >>>> >>>> Stefan Behnel wrote: >>>>> >>>>> you'd call "cython" on a package and it would output a directory with a >>>>> single __init__.so that contains the modules compiled from all .pyx/.py >>>>> files in that package. Importing the package would then trigger an >>>>> import of >>>>> that __init__.so, which in turn will execute code in its init__init__() >>>>> function to register the other modules. >>>> >>>> I don't think it even has to be a directory with an __init__, >>>> it could just be an ordinary .so file with the name of the >>>> package. >>>> >>>> I just tried an experiment in Python: >>>> >>>> # onefilepackage.py >>>> import new, sys >>>> blarg = new.module("blarg") >>>> blarg.thing = "This is the thing" >>>> sys.modules["onefilepackage.blarg"] = blarg >>>> >>>> and two different ways of importing it: >>>> >>>> >>> from onefilepackage import blarg >>>> >>> blarg >>>> >>>> >>> blarg.thing >>>> 'This is the thing' >>>> >>>> >>> import onefilepackage.blarg >>>> >>> onefilepackage.blarg.thing >>>> 'This is the thing' >>>> >>> >>> I'm hacking around these lines. However, I'm working to maintain >>> different modules in different C compilation units, in order to >>> workaround the obvious issue with duplicated global C symbols. >> >> That should be ok as a first attempt to get it working quickly. I'd still >> like to see the modules merged in the long term in order to increase the >> benefit of the more compact format. They'd all share the same code generator >> and Cython's C internals, C helper functions, constants, builtins, etc., but >> each of them would use a separate (name mangling) scope to keep the visible >> C names separate. > > I was thinking that perhaps we could share declarations common to all > cython modules (compiled with that version of Cython) in a cython.h > header file (which is also imho nicer to maintain than a code.put() in > e.g. ModuleNode.generate_module_preamble), and put it in e.g. > Cython/Includes and set the -I C compiler flag to point to the > Includes directory. The only benefit of an external .h file is that the C compiler could use pre-compiled header files. However, the smallest part of such a file is really common to all Cython modules. Most code is at least conditionally included. > Module-specific functions would still be declared static, of course. > And if users want to ship generated C files to avoid Cython as a > dependency, they could simply ship the header and adjust their > setup.py. That's a drawback compared to the current self-contained C files, IMHO. > If you want to merge modules and the "package-approach" is chosen, it > should have well-defined semantics for in-place builds, and > package/__init__.py is preferred over package.so. So how would you > solve that problem without either placing package.so in the package > itself, or giving it another name (and perhaps star-importing it from > __init__.py)? Both would be acceptable, IMHO. It's common to prefix C modules with an underscore, and __init__.py could do the right thing, in one way or another. > Basically, if people want to combine several modules > into one they could use the 'include' statement. > > e.g. in spam.pyx you 'include "ham.pyx"' and in spam.pxd you 'include > "ham.pxd"'. > > (although you'd probably rename ham.pyx to ham.pxi, and you'd probably > merge spam.pxd with ham.pxd) > > In any case, I'm just wondering, would this functionality be more > useful than our current include statement and a cython.h header that > is shared by default? The "include" statement doesn't give you separate namespaces. You'd get a joined module at the Cython code level, which is likely not desirable. However, I admit that you could get a similar 'module' layout by providing an appropriate __init__.py that imports together and/or registering the names from the modules in the package. Stefan From stefan_ml at behnel.de Thu Mar 3 10:59:59 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 03 Mar 2011 10:59:59 +0100 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> <4D6F3887.3090906@behnel.de> Message-ID: <4D6F669F.2040602@behnel.de> Vitja Makarov, 03.03.2011 10:48: > To share common sources is a good idea, we can also share "code" in > libcython-.so > But then we should handle ABI compatibility problems. There is little constant code overlap between modules, and putting that into a separate library would lead to performance regressions (no more inlining etc.). Stefan From wking at drexel.edu Thu Mar 3 13:13:02 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 3 Mar 2011 07:13:02 -0500 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: References: <4D60DE3C.3070306@behnel.de> <4D618506.6070702@behnel.de> <20110303015446.GA27504@tyr.home.net> Message-ID: <20110303121302.GA24387@tyr.home.net> On Wed, Mar 02, 2011 at 06:08:12PM -0800, Robert Bradshaw wrote: > On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King wrote: > > Previous discussions in this thread [1,2] have discussed the issues > > associated with overloading the 'public' keyword. For an example of > > the difficulties this causes, see the ugly workaround [3] in my recent > > commit [4]. Definately worth fixing this syntax. > > > > How do syntax changes with deprecations work? The Parsing module > > doesn't seem to be setup to handle things like that. > > It's not. > > We rarely deprecate syntax, and we should at least give people a > chance to move away from the old syntax first to the new first, so > we'll have to support both for a bit at least. When we do deprecate > the old way of doing things, I think we'll just use > Errors.warning(...). But how would you know which syntax version the source file was aiming for? There should probably be a way to explicitly specify a Cython syntax version, probably from the command line and/or setup.py file, e.g. pyrex_syntax_version = (, , ) as an argument to Cython.Distutils.extension.Extension, which would select the appropriate parser. The default (if the version was not explicitly set), would be to use the current parser. With this setup, projects would only be forced to upgrade if Cython removed support for their syntax entirely, and they could choose to upgrade whenever was most convienient for them. I think the syntax version's major/minor number should probably match the Cython version in which they were introduced. Parser fixes and compatible extensions would bump the number, and you'd probably want to deprecate buggy versions more quickly. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Thu Mar 3 19:54:00 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 3 Mar 2011 10:54:00 -0800 Subject: [Cython] [cython-users] Cython .pxd introspection: listing defined constants In-Reply-To: <20110303121302.GA24387@tyr.home.net> References: <4D60DE3C.3070306@behnel.de> <4D618506.6070702@behnel.de> <20110303015446.GA27504@tyr.home.net> <20110303121302.GA24387@tyr.home.net> Message-ID: On Thu, Mar 3, 2011 at 4:13 AM, W. Trevor King wrote: > On Wed, Mar 02, 2011 at 06:08:12PM -0800, Robert Bradshaw wrote: >> On Wed, Mar 2, 2011 at 5:54 PM, W. Trevor King wrote: >> > Previous discussions in this thread [1,2] have discussed the issues >> > associated with overloading the 'public' keyword. ?For an example of >> > the difficulties this causes, see the ugly workaround [3] in my recent >> > commit [4]. ?Definately worth fixing this syntax. >> > >> > How do syntax changes with deprecations work? ?The Parsing module >> > doesn't seem to be setup to handle things like that. >> >> It's not. >> >> We rarely deprecate syntax, and we should at least give people a >> chance to move away from the old syntax first to the new first, so >> we'll have to support both for a bit at least. When we do deprecate >> the old way of doing things, I think we'll just use >> Errors.warning(...). > > But how would you know which syntax version the source file was aiming > for? ?There should probably be a way to explicitly specify a Cython > syntax version, probably from the command line and/or setup.py file, > e.g. > > ?pyrex_syntax_version = (, , ) > > as an argument to Cython.Distutils.extension.Extension, which would > select the appropriate parser. ?The default (if the version was not > explicitly set), would be to use the current parser. ?With this setup, > projects would only be forced to upgrade if Cython removed support for > their syntax entirely, and they could choose to upgrade whenever was > most convienient for them. > > I think the syntax version's major/minor number should probably match > the Cython version in which they were introduced. ?Parser fixes and > compatible extensions would bump the number, and you'd > probably want to deprecate buggy versions more quickly. With the single exception of introducing the cpdef keyword, I can't think of a single instance where Cython's syntax has ever been changed in a backwards incompatible way, so we've never needed to worry about syntax versioning. I don't see anything with the new syntax that's forcing us to be backwards incompatible, and even the local ugliness of supporting both for the time being is better than going down the route of supporting multiple parsers. - Robert From robertwb at math.washington.edu Thu Mar 3 20:14:19 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 3 Mar 2011 11:14:19 -0800 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: <4D6F669F.2040602@behnel.de> References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> <4D6F3887.3090906@behnel.de> <4D6F669F.2040602@behnel.de> Message-ID: On Thu, Mar 3, 2011 at 1:59 AM, Stefan Behnel wrote: > Vitja Makarov, 03.03.2011 10:48: >> >> To share common sources is a good idea, we can also share "code" in >> libcython-.so >> But then we should handle ABI compatibility problems. > > There is little constant code overlap between modules, and putting that into > a separate library would lead to performance regressions (no more inlining > etc.). There are types that would be nice to share, such as the binding C function and generator types. Both common headers and shared object libraries could be produced, shared, managed by the cythonize(...) build command. - Robert From vitja.makarov at gmail.com Thu Mar 3 20:44:34 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 3 Mar 2011 22:44:34 +0300 Subject: [Cython] strange bug in starred target In-Reply-To: References: Message-ID: 2011/3/3 Vitja Makarov : > 2011/3/3 Vitja Makarov : >> This doesn't work: >> def assign(): >> ? ?a, *b = 1,2,3,4,5 >> ? ?return a, b >> >>>>> import x >>>>> x.assign() >> Traceback (most recent call last): >> ?File "", line 1, in >> ?File "x.pyx", line 6, in x.assign (x.c:445) >> ? ?a, *b = 1,2,3,4,5 >> TypeError: Expected list, got int >> >> And this is ok: >> >> def assign(): >> ? ?a, *b = 1,2,3,4,5 >> ? ?return a, b >> ? ?*a, b = 1,2,3,4,5 >> ? ?return a, b >> >>>>> import x >>>>> x.assign() >> (1, [2, 3, 4, 5]) >> >> > > I hope problem is somewhere in map_starred_assignment() > That wasn't as simple as patch is... Ticket on trac http://trac.cython.org/cython_trac/ticket/664 Pull request https://github.com/cython/cython/pull/7 -- vitja. From greg.ewing at canterbury.ac.nz Thu Mar 3 21:04:10 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 04 Mar 2011 09:04:10 +1300 Subject: [Cython] Multiple modules in one compilation unit In-Reply-To: References: <4D6E19FF.1010002@student.matnat.uio.no> <4D6E2079.9000102@behnel.de> <4D6E21F4.5030006@student.matnat.uio.no> <4D6E2B69.7040404@behnel.de> <4D6E642C.9010001@student.matnat.uio.no> <4D6E69E5.8000806@behnel.de> <4D6EDA50.5030807@canterbury.ac.nz> Message-ID: <4D6FF43A.5030506@canterbury.ac.nz> Lisandro Dalcin wrote: > However, in Python > 3 that is not te case (and only for the .so, for .py is the same as in > Py2), the import machinery adds the entry later, after the > finalization of the module init function. Might be an idea to raise this issue on python-dev, to see if there is a reason for this and whether it could be changed. > I'm tempted to workaround > this by setting the entry in sys.modules right after the call to > PyModule_Create() ... What do you think about this? It's worth a try. I can't think of anything it might break offhand -- but then I'm not really up with the play on what's been happening to the importing machinery over the years. -- Greg From robertwb at math.washington.edu Thu Mar 3 21:49:42 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 3 Mar 2011 12:49:42 -0800 Subject: [Cython] strange bug in starred target In-Reply-To: References: Message-ID: On Thu, Mar 3, 2011 at 11:44 AM, Vitja Makarov wrote: > 2011/3/3 Vitja Makarov : >> 2011/3/3 Vitja Makarov : >>> This doesn't work: >>> def assign(): >>> ? ?a, *b = 1,2,3,4,5 >>> ? ?return a, b >>> >>>>>> import x >>>>>> x.assign() >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ?File "x.pyx", line 6, in x.assign (x.c:445) >>> ? ?a, *b = 1,2,3,4,5 >>> TypeError: Expected list, got int >>> >>> And this is ok: >>> >>> def assign(): >>> ? ?a, *b = 1,2,3,4,5 >>> ? ?return a, b >>> ? ?*a, b = 1,2,3,4,5 >>> ? ?return a, b >>> >>>>>> import x >>>>>> x.assign() >>> (1, [2, 3, 4, 5]) >>> >>> >> >> I hope problem is somewhere in map_starred_assignment() >> > > That wasn't as simple as patch is... > > Ticket on trac http://trac.cython.org/cython_trac/ticket/664 > Pull request https://github.com/cython/cython/pull/7 Thanks. Merged. - Robert From robertwb at math.washington.edu Thu Mar 3 22:05:19 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 3 Mar 2011 13:05:19 -0800 Subject: [Cython] "__ipow__ has wrong number of arguments" is back here In-Reply-To: References: Message-ID: On Wed, Mar 2, 2011 at 5:47 PM, Lisandro Dalcin wrote: > On 2 March 2011 17:14, Lisandro Dalcin wrote: >> On 2 March 2011 16:18, Vitja Makarov wrote: >>> Hi! >>> >>> I noticed that this error came back again. >>> >>> https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/ >>> >> >> I've pushed a fix for __ipow__ to take just two arguments, like >> regular Python classes. >> https://github.com/cython/cython/commit/490fc80e6b494f8ed65f78fc2032d856ec75b8ef >> >> But I forgot to update test for ticket #561, now this should fixed: >> https://github.com/cython/cython/commit/829d6485b77b5688671959f840712c01e6886556 >> > > BTW, do you all agree with this change? To summarize, the C signature of ipow takes three arguments, but the third is often/always garbage, so we shouldn't use it? > I expect this change to break some code, particularly in Sage... It'll be a compile time error fixing a potential runtime crash, so I think that's a good move. Unfortunately the sage-build is still broken due to the fixes for http://trac.cython.org/cython_trac/ticket/654 , but I doubt we use ipow much for two reasons: firstly, most of our objects are immutable, and secondly rather than taking a "modulus" as a third argument for power, the way to do this is to power an element of the appropriate ring, e.g. mod(2, 17)*1000. - Robert From dalcinl at gmail.com Thu Mar 3 22:57:28 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 3 Mar 2011 18:57:28 -0300 Subject: [Cython] "__ipow__ has wrong number of arguments" is back here In-Reply-To: References: Message-ID: On 3 March 2011 18:05, Robert Bradshaw wrote: > On Wed, Mar 2, 2011 at 5:47 PM, Lisandro Dalcin wrote: >> >> BTW, do you all agree with this change? > > To summarize, the C signature of ipow takes three arguments, but the > third is often/always garbage, so we shouldn't use it? > Yes. >> I expect this change to break some code, particularly in Sage... > > It'll be a compile time error fixing a potential runtime crash, so I > think that's a good move. OK, -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From wking at drexel.edu Thu Mar 3 23:11:13 2011 From: wking at drexel.edu (W. Trevor King) Date: Thu, 3 Mar 2011 17:11:13 -0500 Subject: [Cython] cdef-enums-stucts-and-unions progress update In-Reply-To: References: <20110222200207.GA14712@tyr.home.net> <20110225014229.GA29938@tyr.home.net> Message-ID: <20110303221112.GA918@tyr.home.net> Done: * Cleaned all overloaded visibilities out of Parsing and Symtab. * Merged current trunk. Todo: * Remove older Symtab interface so I can remove the WTK_* methods. * Activate `shadow` functionality (via AnalyseDeclarationsTransform.visit_CStructOrUnionDefNode()). Feel free to comment via github: https://github.com/wking/cython/commits/cdef-enums-stucts-and-unions I'm not quite sure how this shadow thing is supposed to work, but I'll take a stab at getting it going if a more seasoned Cythonista is not available for comment ;). -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at math.washington.edu Sun Mar 6 06:33:03 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 5 Mar 2011 21:33:03 -0800 Subject: [Cython] Google Summer of Code Message-ID: Google Summer of Code is going on again this year, and though organizations haven't been chosen again, by all indications, it seems likely that the Python foundation will be a strong participant. Under this umbrella, who out there might be interested in mentoring and/or participating as a student? - Robert From ryandesign at macports.org Sun Mar 6 23:12:06 2011 From: ryandesign at macports.org (Ryan Schmidt) Date: Sun, 6 Mar 2011 16:12:06 -0600 Subject: [Cython] Differing Cython-0.14.1.tar.gz files Message-ID: Dear developers of Cython, There are two different files called Cython-0.14.1.tar.gz -- one in http://www.cython.org/release/ and a different one in http://pypi.python.org/packages/source/C/Cython/: http://trac.macports.org/ticket/28682 Which of these is the "real" 0.14.1? By the way, this has happened before, with version 0.13: http://trac.macports.org/ticket/26627 Why don't you release a version of your software as a single distfile that is identical on all servers? This would make life much easier for package managers like MacPorts, and help you and your users by knowing exactly what version of the software is being used. -Ryan Schmidt, for the MacPorts Project From stefan_ml at behnel.de Mon Mar 7 07:54:01 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Mar 2011 07:54:01 +0100 Subject: [Cython] Differing Cython-0.14.1.tar.gz files In-Reply-To: References: Message-ID: <4D748109.4010009@behnel.de> Ryan Schmidt, 06.03.2011 23:12: > There are two different files called Cython-0.14.1.tar.gz -- one in http://www.cython.org/release/ and a different one in http://pypi.python.org/packages/source/C/Cython/: Intersting. Do you mean "different" as in "different content" (i.e. sources etc.), or just as in "md5sum gives different hashes"? > Why don't you release a version of your software as a single distfile > that is identical on all servers? Well, I don't think there is a reason for that, simply because I doubt that it's being done intentionally. I guess it just lacks a process. In any case, I doubt that there are any differences between the tar.gz files, except for file modification times and potentially the creation time of the C sources. The build process is deterministic. IMHO, the best way to make the releases would be to run setup.py sdist register upload to push them to PyPI, and then take the same tar.gz and copy it over to cython.org. In any case, the one on PyPI should always take the lead, as that's what people get when they run easy_install. I also think we should start signing the released archives. This can be done via distutils' "upload" command by passing upload --sign --identity=[e-mail-address-of-key] Stefan From ryandesign at macports.org Mon Mar 7 08:11:25 2011 From: ryandesign at macports.org (Ryan Schmidt) Date: Mon, 7 Mar 2011 01:11:25 -0600 Subject: [Cython] Differing Cython-0.14.1.tar.gz files In-Reply-To: <4D748109.4010009@behnel.de> References: <4D748109.4010009@behnel.de> Message-ID: <937E6DCB-0938-4FB7-8770-30CEE0647EB8@macports.org> On Mar 7, 2011, at 00:54, Stefan Behnel wrote: > Ryan Schmidt, 06.03.2011 23:12: >> There are two different files called Cython-0.14.1.tar.gz -- one in http://www.cython.org/release/ and a different one in http://pypi.python.org/packages/source/C/Cython/: > > Intersting. Do you mean "different" as in "different content" (i.e. sources etc.), or just as in "md5sum gives different hashes"? Different content. >> Why don't you release a version of your software as a single distfile >> that is identical on all servers? > > Well, I don't think there is a reason for that, simply because I doubt that it's being done intentionally. I guess it just lacks a process. > > In any case, I doubt that there are any differences between the tar.gz files, except for file modification times and potentially the creation time of the C sources. The build process is deterministic. It looks to me like the two distfiles were made from two different git revisions, a couple hours apart. The diff between the two unpacked directories is almost 15,000 lines long. See our ticket: https://trac.macports.org/ticket/28682#comment:5 From ryandesign at macports.org Mon Mar 7 08:25:58 2011 From: ryandesign at macports.org (Ryan Schmidt) Date: Mon, 7 Mar 2011 01:25:58 -0600 Subject: [Cython] Differing Cython-0.14.1.tar.gz files In-Reply-To: <937E6DCB-0938-4FB7-8770-30CEE0647EB8@macports.org> References: <4D748109.4010009@behnel.de> <937E6DCB-0938-4FB7-8770-30CEE0647EB8@macports.org> Message-ID: <06B51683-BC32-4EEE-803E-7417EFDC2E1F@macports.org> On Mar 7, 2011, at 01:11, Ryan Schmidt wrote: > On Mar 7, 2011, at 00:54, Stefan Behnel wrote: >> Ryan Schmidt, 06.03.2011 23:12: >>> There are two different files called Cython-0.14.1.tar.gz -- one in http://www.cython.org/release/ and a different one in http://pypi.python.org/packages/source/C/Cython/: >> >> Intersting. Do you mean "different" as in "different content" (i.e. sources etc.), or just as in "md5sum gives different hashes"? > > Different content. I went back to check the 0.13 distfiles as well. The contents there are clearly different as well; the distfile on www.cython.org is 630K while the one on pypi.python.org is 808K. Here, the difference is that the distfile on pypi.python.org contains these additional files vs. the one on www.cython.org: Cython/Compiler/Scanning.c Cython/Compiler/Visitor.c Cython/Plex/Scanners.c Cython/Runtime/refnanny.c Demos/embed/embedded Demos/embed/embedded.c Demos/embed/embedded.o The 0.14 distfiles also differed between the two servers, although only in the composition of the tar.gz file; the contents are identical. From stefan_ml at behnel.de Mon Mar 7 08:54:02 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 07 Mar 2011 08:54:02 +0100 Subject: [Cython] Differing Cython-0.14.1.tar.gz files In-Reply-To: <06B51683-BC32-4EEE-803E-7417EFDC2E1F@macports.org> References: <4D748109.4010009@behnel.de> <937E6DCB-0938-4FB7-8770-30CEE0647EB8@macports.org> <06B51683-BC32-4EEE-803E-7417EFDC2E1F@macports.org> Message-ID: <4D748F1A.5080607@behnel.de> Ryan Schmidt, 07.03.2011 08:25: > I went back to check the 0.13 distfiles as well. The contents there are clearly different as well; the distfile on www.cython.org is 630K while the one on pypi.python.org is 808K. Here, the difference is that the distfile on pypi.python.org contains these additional files vs. the one on www.cython.org: > > Cython/Compiler/Scanning.c > Cython/Compiler/Visitor.c > Cython/Plex/Scanners.c > Cython/Runtime/refnanny.c These are not needed, although they shouldn't hurt. > Demos/embed/embedded > Demos/embed/embedded.c > Demos/embed/embedded.o These shouldn't be in the sdist. I guess they were built during the test run, though. They should be excluded in the MANIFEST.in file. > The 0.14 distfiles also differed between the two servers, although only in the composition of the tar.gz file; the contents are identical. Good to know. Thanks for checking and reporting this. Stefan From robertwb at math.washington.edu Mon Mar 7 17:44:58 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 7 Mar 2011 08:44:58 -0800 Subject: [Cython] Differing Cython-0.14.1.tar.gz files In-Reply-To: <4D748109.4010009@behnel.de> References: <4D748109.4010009@behnel.de> Message-ID: On Sun, Mar 6, 2011 at 10:54 PM, Stefan Behnel wrote: > Ryan Schmidt, 06.03.2011 23:12: >> >> There are two different files called Cython-0.14.1.tar.gz -- one in >> http://www.cython.org/release/ and a different one in >> ?http://pypi.python.org/packages/source/C/Cython/: > > Intersting. Do you mean "different" as in "different content" (i.e. sources > etc.), or just as in "md5sum gives different hashes"? > > >> Why don't you release a version of your software as a single distfile >> that is identical on all servers? > > Well, I don't think there is a reason for that, simply because I doubt that > it's being done intentionally. I guess it just lacks a process. I just downloaded both tarballs and it looks like there were a small number of commits in the cython.org one that weren't on PyPi. I thought I had uploaded and then done sdist, but maybe I messed something up with branches while doing some last tests. My bad, I've copied the PyPi ones to cython.org. Thanks for bringing this to our attention, we'll be more careful about this in the future. > In any case, I doubt that there are any differences between the tar.gz > files, except for file modification times and potentially the creation time > of the C sources. The build process is deterministic. This is what usually happens, though we should re-download to get exactly the same file. > IMHO, the best way to make the releases would be to run > > ? ?setup.py sdist register upload > > to push them to PyPI, and then take the same tar.gz and copy it over to > cython.org. In any case, the one on PyPI should always take the lead, as > that's what people get when they run easy_install. +1. > I also think we should start signing the released archives. This can be done > via distutils' "upload" command by passing > > ? ?upload --sign --identity=[e-mail-address-of-key] Good idea. - Robert From vitja.makarov at gmail.com Tue Mar 8 11:01:04 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 8 Mar 2011 13:01:04 +0300 Subject: [Cython] InlinedGeneratorExpression question Message-ID: Hi! Here is example code: def foo(seq): cdef int x return any(x for x in seq) Here inner x have type int, how does cdef affects nested scope? Is that correct? -- vitja. From stefan_ml at behnel.de Tue Mar 8 11:19:24 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 11:19:24 +0100 Subject: [Cython] InlinedGeneratorExpression question In-Reply-To: References: Message-ID: <4D7602AC.3080506@behnel.de> Vitja Makarov, 08.03.2011 11:01: > Hi! > > Here is example code: > > def foo(seq): > cdef int x > return any(x for x in seq) > > Here inner x have type int, how does cdef affects nested scope? > Is that correct? Yes, that's intended. Otherwise there'd be no way to assign types to variables used in genexpr/comprehension scopes. The rule is: if there is a variable of the same name cdef-ed in the outer scope, inherit its type. Arguably, this leaves a dangling unused variable in the outer scope, but I find that a minor issue compared to the problem of declaring a type for the inner scope. Stefan From markflorisson88 at gmail.com Tue Mar 8 11:34:01 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 8 Mar 2011 11:34:01 +0100 Subject: [Cython] OpenMP support Message-ID: I'd like to implement OpenMP support for Cython. Looking at http://wiki.cython.org/enhancements/openmp I agree a close 1:1 mapping would be nice. It would probably make sense to start with support for 'nogil' sections because GIL-holding sections would be hard to deal with considering all the 'goto's that are generated (because you may not leave the OpenMP block). See this thread http://comments.gmane.org/gmane.comp.python.cython.devel/8695 for a previous discussion. Looking also at http://wiki.cython.org/enhancements/parallel , for the 'parallel for' support I think the best syntax would be to introduce a fake 'cython.openmp' module as Dag suggested. I propose to start with the following syntax: from python (c)import openmp with nogil: with openmp.parallel('sections'): with openmp.section(): ... I'd like to support the directives as part of the parallel directive, and the stand-alone counterparts. So the equivalent of the above would read: with nogil: with openmp.parallel(): with openmp.sections(): with openmp.section(): ... Perhaps to apply an OpenMP directive to a single statement instead of a structured block, we could follow OpenMP's approach, i.e. allow writing the following instead of its 'with' equivalent: openmp.task(untied=True) a = 1 For the 'for' and 'parallel for' constructs I propose an openmp.range() and openmp.prange() respectively, i.e. with openmp.parallel(): for i in openmp.range(..., firstprivate=('a', 'b'), reduction='+:result'): ... could be written for i in openmp.prange(..., firstprivate=('a', 'b'), reduction='+:result'): ... for short. In the previous thread Sturla mentioned that perhaps it would be easier to implement OpenMP support manually instead of generating OpenMP pragmas and relying on OpenMP compiler support. However, considering that OpenMP has quite a few constructs, not to mention the functionality provided by the OpenMP library itself (all the omp_* functions), I think it will be easier to go the "pragma route", especially if we choose to provide all (or a substantial part) of the OpenMP functionality. Then the OpenMP runtime library (e.g. libgomp when using gcc) can simply be wrapped with an appropriate .pxd in Cython/Includes. What do you guys think? From sturla at molden.no Tue Mar 8 15:52:32 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 15:52:32 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: Message-ID: <4D7642B0.6050702@molden.no> Den 08.03.2011 11:34, skrev mark florisson: > could be written > > for i in openmp.prange(..., firstprivate=('a', 'b'), reduction='+:result'): > ... How would you deal with name mangling, aliases, unboxing of numpy arrays, etc.? Sturla From sturla at molden.no Tue Mar 8 15:54:15 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 15:54:15 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: Message-ID: <4D764317.3020502@molden.no> Den 08.03.2011 11:34, skrev mark florisson: > What do you guys think? Make Cython fully support closures, then we can easily implement our own "OpenMP" using Python threads. No change to the compiler is needed. You might not realise this at first, but OpenMP is just a way of implementing closures in C. Sematically the parallel blocks are just closures executed by multiple threads. Sturla From sturla at molden.no Tue Mar 8 15:46:20 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 15:46:20 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: Message-ID: <4D76413C.4040501@molden.no> Den 08.03.2011 11:34, skrev mark florisson: > What do you guys think? Make Cython fully support closures, then we can easily implement our own "OpenMP" using Python threads. No change to the compiler is needed. You might not realise this at first, but OpenMP is just a way of implementing closures in C. Sematically the parallel blocks are just closures executed by multiple threads. Sturla From d.s.seljebotn at astro.uio.no Tue Mar 8 16:28:12 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Tue, 08 Mar 2011 16:28:12 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: Message-ID: <4D764B0C.6070607@student.matnat.uio.no> On 03/08/2011 11:34 AM, mark florisson wrote: > I'd like to implement OpenMP support for Cython. Looking at Great news! It looks like this will be a topic on the coming workshop, with Francesc coming as well (but nothing wrong with getting started before then). (And please speak up if you are in the right group for a GSoC.) > http://wiki.cython.org/enhancements/openmp I agree a close > 1:1 mapping would be nice. It would probably make sense to start with > support for 'nogil' sections because GIL-holding > sections would be hard to deal with considering all the 'goto's that > are generated (because you may not leave the > OpenMP block). See this thread > http://comments.gmane.org/gmane.comp.python.cython.devel/8695 for a > previous discussion. > > Looking also at http://wiki.cython.org/enhancements/parallel , for the > 'parallel for' support I think the best syntax would > be to introduce a fake 'cython.openmp' module as Dag suggested. I > propose to start with the following syntax: > > from python (c)import openmp > > with nogil: > with openmp.parallel('sections'): > with openmp.section(): I've changed my opinion a little bit since that thread; I'm now rather undecided. Pros: - Easy to get results fast (nothing wrong about Sturla's approach, but realistically it *will* take longer to make it work) - OpenMP already has significant mindshare. Saying "Cython supports OpenMP" will sound soothing to scientific Fortran and C programmers, even if it is technically inferior to other solutions we could find. Cons: - Like Sturla says, closures maps this better in the Cython language (having "private=('a', 'b')" as a syntax for referring to the local variables a and b is rather ugly) - The goto restriction that makes exception handling harder I think that long-term we must find some middle ground that sits between just having the GIL and not having it. E.g., find some way of raising an exception from a nogil block, and also allow more simple code to "get more data to work on" within the context of a single thread. So what you're saying about goto's making me lean against OpenMP. But, perfect is the enemy of good. I certainly won't vote against having this implemented -- because *anything* is better than nothing, and nothing prevents us from building something better in addition to or on top of an initial OpenMP implementation. BTW, I found this framework interesting ... it cites some other frameworks as well (cilk, OpenMP, Intel Threading Blocks), which it beats in a specific situation. http://calvados.di.unipi.it/dokuwiki/doku.php?id=ffnamespace:about I'm NOT saying we should "support FastFlow instead". What I'm thinking is that perhaps the place to start is to ask ourselves: What are the current obstacles to using parallelization frameworks written for C or C++ with Cython? How would one use Cython with these, and do it in a practical way (i.e. not just C code in a nogil block without even exception handling). I.e. what I'd like to tackle first (and perhaps on the workshop) is how to even make it a somewhat pleasant experience to use pthreads or Python threading. Set signal handlers for SIGTERM and friends, implement some form of "delayed" exception creation so that we first pop the stack, acquire the GIL, and *then* construct the exception object... But this is perhaps orthogonal to an OpenMP effort. Dag Sverre From sturla at molden.no Tue Mar 8 16:44:40 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 8 Mar 2011 16:44:40 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7646EB.3020808@molden.no> References: <4D7646EB.3020808@molden.no> Message-ID: <59C321A9-7483-41CE-8100-FA0E7D50B7DF@molden.no> Reposting this, seems it got lost in cyber space. Sturla Den 8. mars 2011 kl. 16.10 skrev Sturla Molden : > Den 08.03.2011 11:34, skrev mark florisson: >> However, considering that OpenMP has quite a few constructs, > > No, OpenMP has very few contructs, not quite a few. And most of them > are not needed, nor wanted, because the functionality is covered by > the Python language (such as scoping rules). > > I.e. we do not need -- nor want -- things like private, shared, > firstprivate or lastprivate in Cython, because the scope of > variables inside closures are already covered by Python syntax. > OpenMP is just making up for the lack of closures in C here. > > We don't need to implement #pragma omp critical, because we have > threading.Lock. > > We don't need to implement #pragma omp atomic, because we have the > GIL (as well as OS and compiler support for atomic operations). > > #pragma omp barrier becomes a butterfly of threading.Event. > > Etc. > > There is not really much left to implement. We need a decorator to > launch closures as parallel blocks (trivial) and classes to do > static, dynamic and guided load balancing (trivial as well). > > It is hardly a page of code to write, just a handful of small > classes, once Cython has closures working properly. > > Sturla > > > > > > > > > > > > > From sturla at molden.no Tue Mar 8 16:10:35 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 16:10:35 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: Message-ID: <4D7646EB.3020808@molden.no> Den 08.03.2011 11:34, skrev mark florisson: > However, considering that OpenMP has quite a few constructs, No, OpenMP has very few contructs, not quite a few. And most of them are not needed, nor wanted, because the functionality is covered by the Python language (such as scoping rules). I.e. we do not need -- nor want -- things like private, shared, firstprivate or lastprivate in Cython, because the scope of variables inside closures are already covered by Python syntax. OpenMP is just making up for the lack of closures in C here. We don't need to implement #pragma omp critical, because we have threading.Lock. We don't need to implement #pragma omp atomic, because we have the GIL (as well as OS and compiler support for atomic operations). #pragma omp barrier becomes a butterfly of threading.Event. Etc. There is not really much left to implement. We need a decorator to launch closures as parallel blocks (trivial) and classes to do static, dynamic and guided load balancing (trivial as well). It is hardly a page of code to write, just a handful of small classes, once Cython has closures working properly. Sturla From stefan_ml at behnel.de Tue Mar 8 17:04:11 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 17:04:11 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7646EB.3020808@molden.no> References: <4D7646EB.3020808@molden.no> Message-ID: <4D76537B.6080804@behnel.de> Sturla Molden, 08.03.2011 16:10: > once Cython has closures working properly. Could you elaborate what you are referring to here? Cython does have closure support. Maybe you are missing closures for cdef functions? Or is it block level closures that you want? That could be done as well, just like we support Py3 scoped list comprehensions etc. I'd dislike the added scoping complexity for them, though. I don't even think cdef function closures are required. You can do the same with a def function and a "with nogil" block. Stefan From markflorisson88 at gmail.com Tue Mar 8 17:10:20 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 8 Mar 2011 17:10:20 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7646EB.3020808@molden.no> References: <4D7646EB.3020808@molden.no> Message-ID: On 8 March 2011 16:10, Sturla Molden wrote: > Den 08.03.2011 11:34, skrev mark florisson: >> >> However, considering that OpenMP has quite a few constructs, > > No, OpenMP has very few contructs, not quite a few. And most of them are not > needed, nor wanted, because the functionality is covered by the Python > language (such as scoping rules). > > I.e. we do not need -- nor want -- things like private, shared, firstprivate > or lastprivate in Cython, because the scope of variables inside closures are > already covered by Python syntax. OpenMP is just making up for the lack of > closures in C here. > > We don't need to implement #pragma omp critical, because we have > threading.Lock. > > We don't need to implement #pragma omp atomic, because we have the GIL (as > well as OS and compiler support for atomic operations). > > #pragma omp barrier becomes a butterfly of threading.Event. > > Etc. But how useful is it to parallelize CPU-bound code while holding GIL? Or do you mean to run the CPU-intensive section in a 'with nogil' block and when you need to do locking, or when you need to deal with Python objects you reaqcuire the GIL? The point of OpenMP is convenience, i.e., having your CPU-bound algorithm parallelized with just a few annotations. If you rewrite your code as a closure for say, a parallel for construct, you'd have to call your closure at every iteration. And then you still have to take care of any reduction and corresponding synchronization (unless you have the GIL already). And then there's still the issue of ordered, single and master constructs. Of course, using threads manually is always possible, it's just not very convenient. > There is not really much left to implement. We need a decorator to launch > closures as parallel blocks (trivial) and classes to do static, dynamic and > guided load balancing (trivial as well). > > It is hardly a page of code to write, just a handful of small classes, once > Cython has closures working properly. What do you mean with proper closures? We have closure support right now. > Sturla > > There is a lot of whitespace at the end of most of your messages. > > > > > > > > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Tue Mar 8 17:32:31 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 17:32:31 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D7646EB.3020808@molden.no> Message-ID: <4D765A1F.5060001@behnel.de> mark florisson, 08.03.2011 17:10: > But how useful is it to parallelize CPU-bound code while holding GIL? > Or do you mean to run the CPU-intensive section in a 'with nogil' > block and when you need to do locking, or when you need to deal with > Python objects you reaqcuire the GIL? You don't need the GIL in order to use Python's C-level locking mechanism. You won't have access to Event, RLock, etc., though, as they are Python API classes. Stefan From markflorisson88 at gmail.com Tue Mar 8 17:33:47 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 8 Mar 2011 17:33:47 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D764B0C.6070607@student.matnat.uio.no> References: <4D764B0C.6070607@student.matnat.uio.no> Message-ID: On 8 March 2011 16:28, Dag Sverre Seljebotn wrote: > On 03/08/2011 11:34 AM, mark florisson wrote: >> >> I'd like to implement OpenMP support for Cython. Looking at > > Great news! It looks like this will be a topic on the coming workshop, with > Francesc coming as well (but nothing wrong with getting started before > then). > > (And please speak up if you are in the right group for a GSoC.) I would love to participate, the problem is that I'm really not sure whether it will fit in my schedule. I will certainly figure this out. >> http://wiki.cython.org/enhancements/openmp I agree a close >> 1:1 mapping would be nice. It would probably make sense to start with >> support for 'nogil' sections because GIL-holding >> sections would be hard to deal with considering all the 'goto's that >> are generated (because you may not leave the >> OpenMP block). See this thread >> http://comments.gmane.org/gmane.comp.python.cython.devel/8695 for a >> previous discussion. >> >> Looking also at http://wiki.cython.org/enhancements/parallel , for the >> 'parallel for' support I think the best syntax would >> be to introduce a fake 'cython.openmp' module as Dag suggested. I >> propose to start with the following syntax: >> >> ? ? from python (c)import openmp >> >> ? ? with nogil: >> ? ? ? ? with openmp.parallel('sections'): >> ? ? ? ? ? ? with openmp.section(): > > I've changed my opinion a little bit since that thread; I'm now rather > undecided. > > Pros: > > ?- Easy to get results fast (nothing wrong about Sturla's approach, but > realistically it *will* take longer to make it work) > ?- OpenMP already has significant mindshare. Saying "Cython supports OpenMP" > will sound soothing to scientific Fortran and C programmers, even if it is > technically inferior to other solutions we could find. > > Cons: > > ?- Like Sturla says, closures maps this better in the Cython language > (having "private=('a', 'b')" as a syntax for referring to the local > variables a and b is rather ugly) > ?- The goto restriction that makes exception handling harder With OpenMP code, exactly how common are exceptions and error handling? I agree that listing privates that way doesn't look very appealing. Maybe 'private=(a, b)' is somewhat better. Or perhaps we should really divert to syntax like cython.openmp("parallel sections private(a, b)"): ... and implement a little OpenMP pragma parser (which shouldn't be very hard). > I think that long-term we must find some middle ground that sits between > just having the GIL and not having it. E.g., find some way of raising an > exception from a nogil block, and also allow more simple code to "get more > data to work on" within the context of a single thread. So what you're > saying about goto's making me lean against OpenMP. > > But, perfect is the enemy of good. I certainly won't vote against having > this implemented -- because *anything* is better than nothing, and nothing > prevents us from building something better in addition to or on top of an > initial OpenMP implementation. > > BTW, I found this framework interesting ... ?it cites some other frameworks > as well (cilk, OpenMP, Intel Threading Blocks), which it beats in a specific > situation. > > http://calvados.di.unipi.it/dokuwiki/doku.php?id=ffnamespace:about > > I'm NOT saying we should "support FastFlow instead". What I'm thinking is > that perhaps the place to start is to ask ourselves: What are the current > obstacles to using parallelization frameworks written for C or C++ with > Cython? How would one use Cython with these, and do it in a practical way > (i.e. not just C code in a nogil block without even exception handling). I think OpenMP is not really designed for error handling, so if we'd go the OpenMP route I think the only way to support exceptions would be to save a pending exception, and "raise" it after the construct ends (in the sequential part), just outside the nogil block (in which case it makes sense to make directives implicitly nogil). Perhaps we can allow GIL acquisition in master constructs and as you mentioned on the wiki, default critical constructs. Exceptions would of course remain pending. Perhaps it's possible to guard against that, i.e., finish the OpenMp construct but skip the actual work in case of a pending exception. I must say, it doesn't sound very promising. So I think we should also ask ourselves what users want, i.e., do they usually want to deal with Python objects in their worksharing code, or do they often want to deal with raw C data? I don't know the answer but I'm guessing they want both. But if users want to deal with Python objects, they could use multiprocessing (although if they wanted to share a lot of data they might have to deal with pickling overhead). > I.e. what I'd like to tackle first (and perhaps on the workshop) is how to > even make it a somewhat pleasant experience to use pthreads or Python > threading. Set signal handlers for SIGTERM and friends, implement some form > of "delayed" exception creation so that we first pop the stack, acquire the > GIL, and *then* construct the exception object... > > But this is perhaps orthogonal to an OpenMP effort. > > Dag Sverre > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From sturla at molden.no Tue Mar 8 17:38:46 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 17:38:46 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D7646EB.3020808@molden.no> Message-ID: <4D765B96.8080502@molden.no> Den 08.03.2011 17:10, skrev mark florisson: > But how useful is it to parallelize CPU-bound code while holding GIL? > Or do you mean to run the CPU-intensive section in a 'with nogil' > block and when you need to do locking, or when you need to deal with > Python objects you reaqcuire the GIL? The Python C API is not thread-safe, thus we cannot allow threads concurrent access. It does not matter if we use the GIL or something else as mutex. A user-space spinlock would probably be faster than kernel objects like the GIL. But that is just implementation details. > The point of OpenMP is convenience, i.e., having your CPU-bound > algorithm parallelized with just a few annotations. If you rewrite > your code as a closure for say, a parallel for construct, you'd have > to call your closure at every iteration. No, that would be hidden away with a decorator. for i in range(n): becomes @openmp.parallel_range(n) def loop(i): > And then you still have to > take care of any reduction and corresponding synchronization (unless > you have the GIL already). And then there's still the issue of > ordered, single and master constructs. Yes, and this is not difficult to implement. Ordered can be implemented with a Queue, master is just a check on thread id. Single can be implemented with an atomic CAS operation. This is just a line or two of library code each. > Of course, using threads manually is always possible, it's just not > very convenient. No it's not, but I am not taking about that. I am talking about how to best map OpenMP to Python. Closure is one method, another that might be possible is a context manager (with-statement). I am not sure if this would be doable or not: with OpenMP( private=(i,j,k), shared=(x,y,z) ) as openmp: instead of #pragma omp parallel. But should we care if this is implemented with OpenMP or Python threads? It's just an implementation detail in the library, not visible to the user. Also I am not against OpenMP, I use it all the time in Fortran :-) Another problem with using OpenMP inside the compiler, as opposed to an external library, is that it depends on a stabile ABI. If an ABI change to Cython's generated C code is made, even a minor change, OpenMP support will be broken. Sturla From sturla at molden.no Tue Mar 8 17:50:00 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 17:50:00 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D76537B.6080804@behnel.de> References: <4D7646EB.3020808@molden.no> <4D76537B.6080804@behnel.de> Message-ID: <4D765E38.5000409@molden.no> Den 08.03.2011 17:04, skrev Stefan Behnel: > Could you elaborate what you are referring to here? Nothing, except my own ignorance :-) As for OpenMP I'd like to add that closures in Cython/Python more cleanly map to Apple's "grand central dispatch" than OpenMP. Yet a third way to easy parallel computing is OpenCL. OpenCL can run on GPU or CPU, depending on driver. Python (or Cython) is a fantastic language for OpenCL, particularly using NumPy and PyOpenCL. Support for anonymous blocks would clean up the syntax if we try to support OpenMP (either way), but Python BDFL hates then (many have asked). It could be a decorator with a or a decorator over a for-loop. Sturla From markflorisson88 at gmail.com Tue Mar 8 18:00:18 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 8 Mar 2011 18:00:18 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D765B96.8080502@molden.no> References: <4D7646EB.3020808@molden.no> <4D765B96.8080502@molden.no> Message-ID: On 8 March 2011 17:38, Sturla Molden wrote: > Den 08.03.2011 17:10, skrev mark florisson: >> >> But how useful is it to parallelize CPU-bound code while holding GIL? >> Or do you mean to run the CPU-intensive section in a 'with nogil' >> block and when you need to do locking, or when you need to deal with >> Python objects you reaqcuire the GIL? > > The Python C API is not thread-safe, thus we cannot allow threads concurrent > access. > > It does not matter if we use the GIL or something else as mutex. A > user-space spinlock would probably be faster than kernel objects like the > GIL. But that is just implementation details. Right, but if you want to deal with Python objects, then you can't just use another lock then the GIL, because there might still be other Python threads. But perhaps you were talking of non-python-object synchronization in nogil blocks. >> The point of OpenMP is convenience, i.e., having your CPU-bound >> algorithm parallelized with just a few annotations. If you rewrite >> your code as a closure for say, a parallel for construct, you'd have >> to call your closure at every iteration. > > No, that would be hidden away with a decorator. > > for i in range(n): > > > becomes > > @openmp.parallel_range(n) > def loop(i): > > Sure, that's not what I was hinting at. What I meant was that the wrapper returned by the decorator would have to call the closure for every iteration, which introduces function call overhead. > >> And then you still have to >> take care of any reduction and corresponding synchronization (unless >> you have the GIL already). And then there's still the issue of >> ordered, single and master constructs. > > Yes, and this is not difficult to implement. Ordered can be implemented with > a Queue, master is just a check on thread id. Single can be implemented with > an atomic CAS operation. This is just a line or two of library code each. >> >> Of course, using threads manually is always possible, it's just not >> very convenient. > > No it's not, but I am not taking about that. I am talking about how to best > map OpenMP to Python. > > Closure is one method, another that might be possible is a context manager > (with-statement). I am not sure if this would be doable or not: > > with OpenMP( private=(i,j,k), shared=(x,y,z) ) as openmp: > > > instead of #pragma omp parallel. > > But should we care if this is implemented with OpenMP or Python threads? > It's just an implementation detail in the library, not visible to the user. Indeed. I guess we just have to establish what we want to do: do we want to support code with Python objects (and exceptions etc), or just C code written in Cython? If it's the latter, then I still think using OpenMP directly would be easier to implement and more convenient for the user than decorators with closures, but maybe I'm too focussed on OpenMP. And indeed, the syntax I proposed (apart from maybe cython.openmp.(p)range) does not look very attractive. > Also I am not against OpenMP, I use it all the time in Fortran :-) > > Another problem with using OpenMP inside the compiler, as opposed to an > external library, is that it depends on a stabile ABI. If an ABI change to > Cython's generated C code is made, even a minor change, OpenMP support will > be broken. > > > Sturla > > > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From sturla at molden.no Tue Mar 8 18:02:13 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 18:02:13 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D764B0C.6070607@student.matnat.uio.no> Message-ID: <4D766115.4000000@molden.no> Den 08.03.2011 17:33, skrev mark florisson: > With OpenMP code, exactly how common are exceptions and error handling? Error handling is always needed, but OpenMP does not help with this. One can use an integer variable as error flag, and use an atomic write in case of error. Sturla From stefan_ml at behnel.de Tue Mar 8 18:03:43 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 18:03:43 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D765E38.5000409@molden.no> References: <4D7646EB.3020808@molden.no> <4D76537B.6080804@behnel.de> <4D765E38.5000409@molden.no> Message-ID: <4D76616F.60606@behnel.de> Sturla Molden, 08.03.2011 17:50: > Den 08.03.2011 17:04, skrev Stefan Behnel: >> Could you elaborate what you are referring to here? > > Nothing, except my own ignorance :-) Now that we've settled that... ;) What about writing up a little "parallel computing with Cython closures" tutorial in the Wiki? BTW, @participants: that would also make a really cool talk for the Open Cython Day at the workshop! Stefan From sturla at molden.no Tue Mar 8 18:27:45 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 18:27:45 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D7646EB.3020808@molden.no> <4D765B96.8080502@molden.no> Message-ID: <4D766711.1090804@molden.no> Den 08.03.2011 18:00, skrev mark florisson: > Sure, that's not what I was hinting at. What I meant was that the > wrapper returned by the decorator would have to call the closure for > every iteration, which introduces function call overhead. OpenMP does this too, in addition to work-scheduling overhead. You don't see it in your C code, but it's there in the generated binary. > Indeed. I guess we just have to establish what we want to do: do we > want to support code with Python objects (and exceptions etc), or just > C code written in Cython? If it's the latter, then I still think using > OpenMP directly would be easier to implement and more convenient for > the user than decorators with closures, but maybe I'm too focussed on > OpenMP. I can probably think of a hundred reasons why we want "OpenMP" in Cython and Python apart from multicore CPUs. It all comes down to the fact that threading.Thread, multiprocessing.Process, Java threads, pthreads, win32 threads, .NET threads, etc., are not abstractions for concurrency that fits well with the human brain. OpenMP is a nice way to "think parallel", not just for compute intensive code, but also I/O bound code: For example listen on a socket for a connection, then handle the connection. Then, put in some pragmas in the code (#pragma omp parallel sections, nowait, etc.), and you have a multithreaded server. Almost all the nastyness of parallel programming is deferred to the compiler. We can get concurrency by putting markups and small hints in the code, instead of writing code to manage everything. Sturla From stefan_ml at behnel.de Tue Mar 8 18:50:15 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 18:50:15 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D7646EB.3020808@molden.no> <4D765B96.8080502@molden.no> Message-ID: <4D766C57.2090806@behnel.de> mark florisson, 08.03.2011 18:00: > What I meant was that the > wrapper returned by the decorator would have to call the closure for > every iteration, which introduces function call overhead. >[...] > I guess we just have to establish what we want to do: do we > want to support code with Python objects (and exceptions etc), or just > C code written in Cython? I like the approach that Sturla mentioned: using closures to implement worker threads. I think that's very pythonic. You could do something like this, for example: def worker(): for item in queue: with nogil: do_stuff(item) queue.extend(work_items) start_threads(worker, count) Note that the queue is only needed to tell the thread what to work on. A lot of things can be shared over the closure. So the queue may not even be required in many cases. Stefan From hoytak at stat.washington.edu Tue Mar 8 18:58:21 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Tue, 8 Mar 2011 09:58:21 -0800 Subject: [Cython] OpenMP support In-Reply-To: <4D766711.1090804@molden.no> References: <4D7646EB.3020808@molden.no> <4D765B96.8080502@molden.no> <4D766711.1090804@molden.no> Message-ID: Apologies for the OT post, but a quick FYI: I've been using openmp with cython pretty extensively, usually wrapped in static inline or template functions in a separate header. However, I've been hindered by a strange bug that seems to appear when python loads gomp, gcc's openmp implementation (http://bugs.python.org/issue11275). For me, it causes matplatlib to become unusable. I don't know if it's a bug in matplotlib, in python, in gcc, or what -- a better programmer than me could probably track it down -- but it has definitely made working with openmp/cython a pain. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From vitja.makarov at gmail.com Tue Mar 8 20:12:32 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 8 Mar 2011 22:12:32 +0300 Subject: [Cython] unbound local variables Message-ID: What is the right way to handle cdefed unbounds? cdef object foo print foo cdef int foo print foo And how buffers and arrays should be handled? Now I'm skipping buffers, arrays and structs. There are some examples in test suite: ------------------------------------------------------------ ... cdef void spam(): cdef long long L cdef unsigned long long U cdef object x L = x ^ ------------------------------------------------------------ ass2longlong.pyx:5:9: local variable 'x' referenced before assignment cdef void spam(): cdef object x del x[17:42] ^ ------------------------------------------------------------ delslice.pyx:3:9: local variable 'x' referenced before assignment cdef void tomato(): cdef Spam spam cdef SuperSpam superspam spam = superspam ^ ------------------------------------------------------------ extcmethcall.pyx:16:20: local variable 'superspam' referenced before assignment def f(Grail g): cdef int i = 0 cdef Swallow s cdef object x g = x ^ ------------------------------------------------------------ extcoerce.pyx:13:9: local variable 'x' referenced before assignment Should this raise error message or not? -- vitja. From faltet at pytables.org Tue Mar 8 20:13:03 2011 From: faltet at pytables.org (Francesc Alted) Date: Tue, 8 Mar 2011 20:13:03 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D765B96.8080502@molden.no> References: <4D765B96.8080502@molden.no> Message-ID: <201103082013.03330.faltet@pytables.org> A Tuesday 08 March 2011 17:38:46 Sturla Molden escrigu?: > But should we care if this is implemented with OpenMP or Python > threads? It's just an implementation detail in the library, not > visible to the user. > > Also I am not against OpenMP, I use it all the time in Fortran :-) > > Another problem with using OpenMP inside the compiler, as opposed to > an external library, is that it depends on a stabile ABI. If an ABI > change to Cython's generated C code is made, even a minor change, > OpenMP support will be broken. And another problem that should be taken in account is that MS Visual Studio does not offer OpenMP in the Express edition (the free, as in beer, one). And, in addition, we can expect problems with the OpenMP support in GCC for Win. As I see this, if we could stick using just Python threads, that would be our best bet for having a good adoption of future Cython parallel capabilities. In the case that we need dealing with a low-level C-API thread library, I'd use pthreads, with a possible light wrapper for using it from the Windows thread API on Windows machines. -- Francesc Alted From faltet at pytables.org Tue Mar 8 20:16:02 2011 From: faltet at pytables.org (Francesc Alted) Date: Tue, 8 Mar 2011 20:16:02 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D766C57.2090806@behnel.de> References: <4D766C57.2090806@behnel.de> Message-ID: <201103082016.02413.faltet@pytables.org> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigu?: > mark florisson, 08.03.2011 18:00: > > What I meant was that the > > wrapper returned by the decorator would have to call the closure > > for every iteration, which introduces function call overhead. > > > >[...] > > > > I guess we just have to establish what we want to do: do we > > want to support code with Python objects (and exceptions etc), or > > just C code written in Cython? > > I like the approach that Sturla mentioned: using closures to > implement worker threads. I think that's very pythonic. You could do > something like this, for example: > > def worker(): > for item in queue: > with nogil: > do_stuff(item) > > queue.extend(work_items) > start_threads(worker, count) > > Note that the queue is only needed to tell the thread what to work > on. A lot of things can be shared over the closure. So the queue may > not even be required in many cases. I like this approach too. I suppose that you will need to annotate the items so that they are not Python objects, no? Something like: def worker(): cdef int item # tell that item is not a Python object! for item in queue: with nogil: do_stuff(item) queue.extend(work_items) start_threads(worker, count) -- Francesc Alted From sturla at molden.no Tue Mar 8 20:18:10 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 20:18:10 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D766C57.2090806@behnel.de> References: <4D7646EB.3020808@molden.no> <4D765B96.8080502@molden.no> <4D766C57.2090806@behnel.de> Message-ID: <4D7680F2.5000700@molden.no> Den 08.03.2011 18:50, skrev Stefan Behnel: > Note that the queue is only needed to tell the thread what to work on. > A lot of things can be shared over the closure. So the queue may not > even be required in many cases. Instead of putting a "#pragma omp parallel for" over the for loop, we put the for-loop inside a closure. That is about the same amount of extra code, positioned similarly in the code, achieving the same thing. Private variables are passed as calling arguments to the closure, shared variables are shared over the closure. That is Python syntax, so we don't need specifiers for 'private' and 'shared' like OpenMP. We still need sychonization and scheduling primitives similar to those in OpenMP. For example a special 'range' function that will share the workload of a for loop. But this is not a major programming task. What I am trying to say is that the major argument for OpenMP in C and Fortran is lack of closures. That does not apply to Cython anymore (as I happily learned today). That is why I think this is a library and not a compiler issue. I'll make an example on the wiki when I get som spare time. Possibly with an OpenMP example to show the similarty. One really has to look at it side-by-side to see that it's the same. The same arguments holds for Apple's GCD as well. GCD makes the closure semantics explicit by an extension to objective-C. We need to discourage Java inspired inheritance from threading.Thread, and encourage closures instead. And some very clear examples :-) Sturla From sturla at molden.no Tue Mar 8 20:24:51 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 20:24:51 +0100 Subject: [Cython] OpenMP support In-Reply-To: <201103082013.03330.faltet@pytables.org> References: <4D765B96.8080502@molden.no> <201103082013.03330.faltet@pytables.org> Message-ID: <4D768283.4070902@molden.no> Den 08.03.2011 20:13, skrev Francesc Alted: > And another problem that should be taken in account is that MS Visual > Studio does not offer OpenMP in the Express edition (the free, as in > beer, one). Which is why one should get the Windows 7 SDK instead :-) > As I see this, if we could stick using just Python threads, that would > be our best bet for having a good adoption of future Cython parallel > capabilities. Yes, if they are there, we have threads. If they are not, we don't. We don't need to assume anything about the platform, libraries, etc. It's all taken care of by Python. > In the case that we need dealing with a low-level C-API > thread library, I'd use pthreads, with a possible light wrapper for > using it from the Windows thread API on Windows machines. > pthreads are available on Windows as well. Sturla From stefan_ml at behnel.de Tue Mar 8 20:32:22 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 20:32:22 +0100 Subject: [Cython] OpenMP support In-Reply-To: <201103082016.02413.faltet@pytables.org> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> Message-ID: <4D768446.5070300@behnel.de> Francesc Alted, 08.03.2011 20:16: > A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigu?: >> mark florisson, 08.03.2011 18:00: >>> What I meant was that the >>> wrapper returned by the decorator would have to call the closure >>> for every iteration, which introduces function call overhead. >>> >>> [...] >>> >>> I guess we just have to establish what we want to do: do we >>> want to support code with Python objects (and exceptions etc), or >>> just C code written in Cython? >> >> I like the approach that Sturla mentioned: using closures to >> implement worker threads. I think that's very pythonic. You could do >> something like this, for example: >> >> def worker(): >> for item in queue: >> with nogil: >> do_stuff(item) >> >> queue.extend(work_items) >> start_threads(worker, count) >> >> Note that the queue is only needed to tell the thread what to work >> on. A lot of things can be shared over the closure. So the queue may >> not even be required in many cases. > > I like this approach too. I suppose that you will need to annotate the > items so that they are not Python objects, no? Something like: > > def worker(): > cdef int item # tell that item is not a Python object! > for item in queue: > with nogil: > do_stuff(item) > > queue.extend(work_items) > start_threads(worker, count) You *can* use Python references inside of nogil blocks, even if there's a lot of stuff that you can't do with them, such as looking up Python attributes, calling them, or even assigning their reference to another Python variable. But you can access cdef attributes on them, for example, and some simple conversions to C types may also work (AFAIR). But you're generally right that passing it into a function (as in the example above) would likely not work. In any case, Cython will tell you if you try to do anything that's not allowed. Stefan From stefan_ml at behnel.de Tue Mar 8 20:34:57 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 20:34:57 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7680F2.5000700@molden.no> References: <4D7646EB.3020808@molden.no> <4D765B96.8080502@molden.no> <4D766C57.2090806@behnel.de> <4D7680F2.5000700@molden.no> Message-ID: <4D7684E1.8000207@behnel.de> Sturla Molden, 08.03.2011 20:18: > We still need sychonization and scheduling primitives similar to those in > OpenMP. For example a special 'range' function that will share the workload > of a for loop. But this is not a major programming task. ... but that could be put into a builtin Cython library, right? At least a couple of primitives, decorators, helpers, etc. Stefan From stefan_ml at behnel.de Tue Mar 8 20:40:14 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 08 Mar 2011 20:40:14 +0100 Subject: [Cython] unbound local variables In-Reply-To: References: Message-ID: <4D76861E.8080801@behnel.de> Vitja Makarov, 08.03.2011 20:12: > What is the right way to handle cdefed unbounds? > > cdef object foo > print foo > > cdef int foo > print foo Fail with a compile error. > And how buffers and arrays should be handled? Now I'm skipping > buffers, arrays and structs. > > > There are some examples in test suite: > > ------------------------------------------------------------ > ... > cdef void spam(): > cdef long long L > cdef unsigned long long U > cdef object x > L = x > ^ > ------------------------------------------------------------ > > ass2longlong.pyx:5:9: local variable 'x' referenced before assignment > > cdef void spam(): > cdef object x > del x[17:42] > ^ > ------------------------------------------------------------ > > delslice.pyx:3:9: local variable 'x' referenced before assignment > > cdef void tomato(): > cdef Spam spam > cdef SuperSpam superspam > spam = superspam > ^ > ------------------------------------------------------------ > > extcmethcall.pyx:16:20: local variable 'superspam' referenced before assignment > > def f(Grail g): > cdef int i = 0 > cdef Swallow s > cdef object x > g = x > ^ > ------------------------------------------------------------ > > extcoerce.pyx:13:9: local variable 'x' referenced before assignment > > > Should this raise error message or not? Absolutely. The examples above are most likely left-overs from Pyrex' test suite, which wasn't meant to run the code that it generated. Stefan From faltet at pytables.org Tue Mar 8 20:43:34 2011 From: faltet at pytables.org (Francesc Alted) Date: Tue, 8 Mar 2011 20:43:34 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D768283.4070902@molden.no> References: <201103082013.03330.faltet@pytables.org> <4D768283.4070902@molden.no> Message-ID: <201103082043.34165.faltet@pytables.org> A Tuesday 08 March 2011 20:24:51 Sturla Molden escrigu?: > Den 08.03.2011 20:13, skrev Francesc Alted: > > And another problem that should be taken in account is that MS > > Visual Studio does not offer OpenMP in the Express edition (the > > free, as in beer, one). > > Which is why one should get the Windows 7 SDK instead :-) Yeah, but this is largely obscure and not-well documented venue (clearly, Microsoft does not want people going that route). > > In the case that we need dealing with a low-level C-API > > thread library, I'd use pthreads, with a possible light wrapper for > > using it from the Windows thread API on Windows machines. > > pthreads are available on Windows as well. You mean http://sourceware.org/pthreads-win32/index.html? Uh, this is an important dependency, and the project seems not maintained anymore (no updates since 2006). Besides, its LGPL license does not seem a good choice for Cython. I was thinking in something more like: http://locklessinc.com/articles/pthreads_on_windows/ which seems simple enough (just a .h header), and it also comes with a sensible license (BSD). -- Francesc Alted From sturla at molden.no Tue Mar 8 20:50:59 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 08 Mar 2011 20:50:59 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7684E1.8000207@behnel.de> References: <4D7646EB.3020808@molden.no> <4D765B96.8080502@molden.no> <4D766C57.2090806@behnel.de> <4D7680F2.5000700@molden.no> <4D7684E1.8000207@behnel.de> Message-ID: <4D7688A3.1000003@molden.no> Den 08.03.2011 20:34, skrev Stefan Behnel: > > ... but that could be put into a builtin Cython library, right? At > least a couple of primitives, decorators, helpers, etc. Yes. Sturla From greg.ewing at canterbury.ac.nz Wed Mar 9 00:27:20 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 09 Mar 2011 12:27:20 +1300 Subject: [Cython] OpenMP support In-Reply-To: <4D768446.5070300@behnel.de> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D768446.5070300@behnel.de> Message-ID: <4D76BB58.2090305@canterbury.ac.nz> Stefan Behnel wrote: > You *can* use Python references inside of nogil blocks, even if there's > a lot of stuff that you can't do with them, ... But you can access cdef > attributes on them, Strictly speaking it's not even safe to do that, unless you know there's some locking mechanism in effect that prevents the underlying Python object going away from under you. -- Greg From stefan_ml at behnel.de Wed Mar 9 06:26:40 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 09 Mar 2011 06:26:40 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D76BB58.2090305@canterbury.ac.nz> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D768446.5070300@behnel.de> <4D76BB58.2090305@canterbury.ac.nz> Message-ID: <4D770F90.40109@behnel.de> Greg Ewing, 09.03.2011 00:27: > Stefan Behnel wrote: > >> You *can* use Python references inside of nogil blocks, even if there's a >> lot of stuff that you can't do with them, ... But you can access cdef > > attributes on them, > > Strictly speaking it's not even safe to do that, unless > you know there's some locking mechanism in effect that > prevents the underlying Python object going away from > under you. Right. It also depends on where the reference came from. If it's held by a local variable that entered a nogil block, it's safe to use the reference, because it cannot be deallocated inside of the nogil block. However, if it is in a closure or in an object attribute that other parts of the code have access to, they may choose to change the binding, thus potentially deallocating the reference. Maybe Cython could do another tiny bit more here, e.g. issue a minor warning for potential concurrency problems like this, that users could enable in their test suites. Stefan From faltet at pytables.org Wed Mar 9 10:43:01 2011 From: faltet at pytables.org (Francesc Alted) Date: Wed, 9 Mar 2011 10:43:01 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D766115.4000000@molden.no> References: <4D766115.4000000@molden.no> Message-ID: <201103091043.01846.faltet@pytables.org> A Tuesday 08 March 2011 18:02:13 Sturla Molden escrigu?: > Den 08.03.2011 17:33, skrev mark florisson: > > With OpenMP code, exactly how common are exceptions and error > > handling? > > Error handling is always needed, but OpenMP does not help with this. > > One can use an integer variable as error flag, and use an atomic > write in case of error. Yeah. I use exactly this technique for tracking errors in threaded code. It is a bit messy, but when done correctly, it works great. -- Francesc Alted From cross at ueh0.bank.gov.ua Wed Mar 9 12:13:15 2011 From: cross at ueh0.bank.gov.ua (Oleksandr Kreshchenko) Date: Wed, 09 Mar 2011 13:13:15 +0200 Subject: [Cython] Sources list handling in Build/Dependencies.py:create_extension_list Message-ID: <4D7760CB.3040606@ueh0.bank.gov.ua> Hello! Following the "CEP 201 - Distutils Preprocessing" (http://wiki.cython.org/enhancements/distutils_preprocessing) there are two possibilities to build a module with cythonize function in setup.py. Second one uses list of distutils.extension.Extension class which accept all necessary compiler and linker options along with list of non-pyx sources to compile and link with for a particular extension module. As this way appears more natural for me I do:: ext_modules = cythonize([Extension( 'CyCont', ['CyCont.pyx', 'Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c', 'MemMan.c', '../Log/Log.c'], define_macros = [('CONT_API', '')], # None value expands to 1, whereas '' to empty include_dirs = dirs, #libraries = ['libgcov'], extra_compile_args = sys_comp_opts[sys.platform], extra_link_args = sys_link_opts[sys.platform])], show_version = 1, language_level = 3, verbose = 1, include_path = dirs) But, only CyCont.c is created from CyCont.pyx then compiled and linked into the module CyCont.so, but the rest of sources to be wrapped are completely ignored in the build process. So, doing import CyCont I've got undefined symbol error. However when I move 'CyCont.pyx' to the end of the sources list i.e.:: ext_modules = cythonize([Extension( 'CyCont', ['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c', 'MemMan.c', '../Log/Log.c', 'CyCont.pyx'], ... the module is built properly. Execution flow goes though Cython/Build/Dependencies.py (latest revision):: 393 def create_extension_list(patterns, exclude=[], ctx=None, aliases=None) ... 404 for pattern in patterns: 405 if isinstance(pattern, str): 406 filepattern = pattern 407 template = None 408 name = '*' 409 base = None 410 exn_type = Extension 411 elif isinstance(pattern, Extension): 412 filepattern = pattern.sources[0] 413 if os.path.splitext(filepattern)[1] not in ('.py', '.pyx'): 414 # ignore non-cython modules 415 module_list.append(pattern) 416 continue 417 template = pattern 418 name = template.name 419 base = DistutilsInfo(exn=template) 420 exn_type = template.__class__ ... 423 for file in glob(filepattern): ... 437 module_list.append(exn_type( 438 name=module_name, 439 sources=[file], 440 **kwds)) ... 443 return module_list If the first list member (sources[0]) is *.pyx or *.py then the only list member is CyCont.pyx (see line 439). Otherwise if CyCont.pyx is placed at the end of the list or, generally, the first list member is neither *.pyx nor *.py than the loop over patterns/extensions (on line 404) breaks at the line 416 so sources list is left untouched and cythonize function (line 446) does the job. This difference is a bug. Isn't it? Have "sources[0]" on line 412 and "sources=[file]" on line 439 to be revised? BTW, I catch the idea of twiking an extension in setup.py like this:: for ext in ext_modules: ext.define_macros = [('CONT_API', '')] ext.extra_compile_args = sys_comp_opts[sys.platform] ext.extra_link_args = sys_link_opts[sys.platform] ext.sources.extend(['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c', 'MemMan.c', '../Log/Log.c']) but it seams **better** to handle this stuff in the first place within Extension(...), not to mention using module-level distutils directives. Best regards, Alex From robertwb at math.washington.edu Wed Mar 9 19:42:26 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 9 Mar 2011 10:42:26 -0800 Subject: [Cython] Sources list handling in Build/Dependencies.py:create_extension_list In-Reply-To: <4D7760CB.3040606@ueh0.bank.gov.ua> References: <4D7760CB.3040606@ueh0.bank.gov.ua> Message-ID: On Wed, Mar 9, 2011 at 3:13 AM, Oleksandr Kreshchenko wrote: > Hello! > > Following the "CEP 201 - Distutils Preprocessing" > (http://wiki.cython.org/enhancements/distutils_preprocessing) > there are two possibilities to build a module with cythonize function in > setup.py. > > Second one uses list of distutils.extension.Extension class which accept all > necessary compiler and > linker options along with list of non-pyx sources to compile and link with > for a particular extension module. > As this way appears more natural for me I do:: > > ? ?ext_modules = cythonize([Extension( > ? ? ? ?'CyCont', ['CyCont.pyx', 'Cont.c', 'ContCet.c', 'Clash.c', > 'ContInit.c', 'MemMan.c', '../Log/Log.c'], > ? ? ? ?define_macros = [('CONT_API', '')], # None value expands to 1, > whereas '' to empty > ? ? ? ?include_dirs = dirs, #libraries = ['libgcov'], > ? ? ? ?extra_compile_args = sys_comp_opts[sys.platform], > ? ? ? ?extra_link_args = sys_link_opts[sys.platform])], > ? ? ?show_version = 1, language_level = 3, verbose = 1, include_path = dirs) > > But, only CyCont.c is created from CyCont.pyx then compiled and linked into > the module CyCont.so, > but the rest of sources to be wrapped are completely ignored in the build > process. > So, doing import CyCont I've got undefined symbol error. > > However when I move 'CyCont.pyx' to the end of the sources list i.e.:: > > ? ?ext_modules = cythonize([Extension( > ? ? ? ?'CyCont', ['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c', > 'MemMan.c', '../Log/Log.c', 'CyCont.pyx'], > > ? ?... > > the module is built properly. > > Execution flow goes though Cython/Build/Dependencies.py (latest revision):: > > 393 def create_extension_list(patterns, exclude=[], ctx=None, aliases=None) > ... > 404 ? ? for pattern in patterns: > 405 ? ? ? ? if isinstance(pattern, str): > 406 ? ? ? ? ? ? filepattern = pattern > 407 ? ? ? ? ? ? template = None > 408 ? ? ? ? ? ? name = '*' > 409 ? ? ? ? ? ? base = None > 410 ? ? ? ? ? ? exn_type = Extension > 411 ? ? ? ? elif isinstance(pattern, Extension): > 412 ? ? ? ? ? ? filepattern = pattern.sources[0] > 413 ? ? ? ? ? ? if os.path.splitext(filepattern)[1] not in ('.py', '.pyx'): > 414 ? ? ? ? ? ? ? ? # ignore non-cython modules > 415 ? ? ? ? ? ? ? ? module_list.append(pattern) > 416 ? ? ? ? ? ? ? ? continue > 417 ? ? ? ? ? ? template = pattern > 418 ? ? ? ? ? ? name = template.name > 419 ? ? ? ? ? ? base = DistutilsInfo(exn=template) > 420 ? ? ? ? ? ? exn_type = template.__class__ > ... > 423 ? ? ? ? for file in glob(filepattern): > ... > 437 ? ? ? ? ? ? ? ? module_list.append(exn_type( > 438 ? ? ? ? ? ? ? ? ? ? ? ? name=module_name, > 439 ? ? ? ? ? ? ? ? ? ? ? ? sources=[file], > 440 ? ? ? ? ? ? ? ? ? ? ? ? **kwds)) > ... > 443 ? ? return module_list > > If the first list member (sources[0]) is *.pyx or *.py then the only list > member is CyCont.pyx (see line 439). > > Otherwise if CyCont.pyx is placed at the end of the list or, generally, the > first list member is > neither *.pyx nor *.py than the loop over patterns/extensions (on line 404) > breaks at the line 416 > so sources list is left untouched and cythonize function (line 446) does the > job. > > This difference is a bug. Isn't it? Yes, for sure, thanks for the report. https://github.com/cython/cython/commit/0fefeb25858ac943a39c4f9ca7e0a493052517a1 > Have "sources[0]" on line 412 and "sources=[file]" on line 439 to be > revised? > > > BTW, I catch the idea of twiking an extension in setup.py like this:: > > for ext in ext_modules: > ? ?ext.define_macros = [('CONT_API', '')] > ? ?ext.extra_compile_args = sys_comp_opts[sys.platform] > ? ?ext.extra_link_args = sys_link_opts[sys.platform] > ? ?ext.sources.extend(['Cont.c', 'ContCet.c', 'Clash.c', 'ContInit.c', > 'MemMan.c', '../Log/Log.c']) > > but it seams **better** to handle this stuff in the first place within > Extension(...), > not to mention using module-level distutils directives. Yep. The build code is set to handle subclasses of Extension, but there's only so much you can do, otherwise it becomes difficult to use two projects that need different distutils hacks. - Robert From robertwb at math.washington.edu Fri Mar 11 01:46:31 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 10 Mar 2011 16:46:31 -0800 Subject: [Cython] OpenMP support In-Reply-To: <201103082016.02413.faltet@pytables.org> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> Message-ID: On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted wrote: > A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigu?: >> mark florisson, 08.03.2011 18:00: >> > What I meant was that the >> > wrapper returned by the decorator would have to call the closure >> > for every iteration, which introduces function call overhead. >> > >> >[...] >> > >> > I guess we just have to establish what we want to do: do we >> > want to support code with Python objects (and exceptions etc), or >> > just C code written in Cython? >> >> I like the approach that Sturla mentioned: using closures to >> implement worker threads. I think that's very pythonic. You could do >> something like this, for example: >> >> ? ? ?def worker(): >> ? ? ? ? ?for item in queue: >> ? ? ? ? ? ? ?with nogil: >> ? ? ? ? ? ? ? ? ?do_stuff(item) >> >> ? ? ?queue.extend(work_items) >> ? ? ?start_threads(worker, count) >> >> Note that the queue is only needed to tell the thread what to work >> on. A lot of things can be shared over the closure. So the queue may >> not even be required in many cases. > > I like this approach too. ?I suppose that you will need to annotate the > items so that they are not Python objects, no? ?Something like: > > ? ? def worker(): > ? ? ? ? cdef int item ?# tell that item is not a Python object! > ? ? ? ? for item in queue: > ? ? ? ? ? ? with nogil: > ? ? ? ? ? ? ? ? do_stuff(item) > > ? ? queue.extend(work_items) > ? ? start_threads(worker, count) On a slightly higher level, are we just trying to use OpenMP from Cython, or are we trying to build it into the language? If the former, it may make sense to stick closer than one might otherwise be tempted in terms of API to the underlying C to leverage the existing documentation. A library with a more Pythonic interface could perhaps be written on top of that. Alternatively, if we're building it into Cython itself, I'd it might be worth modeling it after the multiprocessing module (though I understand it would be implemented with threads), which I think is a decent enough model for managing embarrassingly parallel operations. The above code is similar to that, though I'd prefer the for loop implicit rather than as part of the worker method (or at least as an argument). If we went this route, what are the advantages of using OpenMP over, say, pthreads in the background? (And could the latter be done with just a library + some fancy GIL specifications?) One thing that's nice about OpenMP as implemented in C is that the serial code looks almost exactly like the parallel code; the code at http://wiki.cython.org/enhancements/openmp has this property too. Also, I like the idea of being able to hold the GIL by the invoking thread and having the "sharing" threads do the appropriate locking among themselves when needed if possible, e.g. for exception raising. Another thought I had is, there might be other usecases for being able to emit generic pragmas statements, how far would that get us? - Robert From stefan_ml at behnel.de Fri Mar 11 08:20:28 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Mar 2011 08:20:28 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> Message-ID: <4D79CD3C.6050004@behnel.de> Robert Bradshaw, 11.03.2011 01:46: > On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted wrote: >> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigu?: >>> mark florisson, 08.03.2011 18:00: >>>> What I meant was that the >>>> wrapper returned by the decorator would have to call the closure >>>> for every iteration, which introduces function call overhead. >>>> >>>> [...] >>>> >>>> I guess we just have to establish what we want to do: do we >>>> want to support code with Python objects (and exceptions etc), or >>>> just C code written in Cython? >>> >>> I like the approach that Sturla mentioned: using closures to >>> implement worker threads. I think that's very pythonic. You could do >>> something like this, for example: >>> >>> def worker(): >>> for item in queue: >>> with nogil: >>> do_stuff(item) >>> >>> queue.extend(work_items) >>> start_threads(worker, count) >>> >>> Note that the queue is only needed to tell the thread what to work >>> on. A lot of things can be shared over the closure. So the queue may >>> not even be required in many cases. >> >> I like this approach too. I suppose that you will need to annotate the >> items so that they are not Python objects, no? Something like: >> >> def worker(): >> cdef int item # tell that item is not a Python object! >> for item in queue: >> with nogil: >> do_stuff(item) >> >> queue.extend(work_items) >> start_threads(worker, count) > > On a slightly higher level, are we just trying to use OpenMP from > Cython, or are we trying to build it into the language? If the former, > it may make sense to stick closer than one might otherwise be tempted > in terms of API to the underlying C to leverage the existing > documentation. A library with a more Pythonic interface could perhaps > be written on top of that. Alternatively, if we're building it into > Cython itself, I'd it might be worth modeling it after the > multiprocessing module (though I understand it would be implemented > with threads), which I think is a decent enough model for managing > embarrassingly parallel operations. +1 > The above code is similar to that, > though I'd prefer the for loop implicit rather than as part of the > worker method (or at least as an argument). It provides a simple way to write per-thread initialisation code, though. And it's likely easier to make looping fast than to speed up the call into a closure. However, eventually, both ways will need to be supported anyway. > If we went this route, > what are the advantages of using OpenMP over, say, pthreads in the > background? (And could the latter be done with just a library + some > fancy GIL specifications?) In the above example, basically everything is explicit and nothing more than a simplified threading setup is needed. Even the implementation of "start_threads()" could be done in a couple of lines of Python code, including the collection of results and errors. If someone thinks we need more than that, I'd like to see a couple of concrete use cases and code examples first. > One thing that's nice about OpenMP as > implemented in C is that the serial code looks almost exactly like the > parallel code; the code at http://wiki.cython.org/enhancements/openmp > has this property too. Writing it with a closure isn't really that much different. You can put the inner function right where it would normally get executed and add a bit of calling/load distributing code below it. Not that bad IMO. It may be worth providing some ready-to-use decorators to do the load balancing, but I don't really like the idea of having a decorator magically invoke the function in-place that it decorates. > Also, I like the idea of being able to hold the GIL by the invoking > thread and having the "sharing" threads do the appropriate locking > among themselves when needed if possible, e.g. for exception raising. I like the explicit "with nogil" block in my example above. It makes it easy to use normal Python setup code, to synchronise based on the GIL if desired (e.g. to use a normal Python queue for communication), and it's simple enough not to get in the way. I think it simplifies things a lot when code can rely on the GIL being held when entering the thread function. Threading is complicated enough to keep it as explicit as possible. Stefan From d.s.seljebotn at astro.uio.no Fri Mar 11 08:56:51 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 11 Mar 2011 08:56:51 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D79CD3C.6050004@behnel.de> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D79CD3C.6050004@behnel.de> Message-ID: <4D79D5C3.7080805@student.matnat.uio.no> On 03/11/2011 08:20 AM, Stefan Behnel wrote: > Robert Bradshaw, 11.03.2011 01:46: >> On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted >> wrote: >>> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigu?: >>>> mark florisson, 08.03.2011 18:00: >>>>> What I meant was that the >>>>> wrapper returned by the decorator would have to call the closure >>>>> for every iteration, which introduces function call overhead. >>>>> >>>>> [...] >>>>> >>>>> I guess we just have to establish what we want to do: do we >>>>> want to support code with Python objects (and exceptions etc), or >>>>> just C code written in Cython? >>>> >>>> I like the approach that Sturla mentioned: using closures to >>>> implement worker threads. I think that's very pythonic. You could do >>>> something like this, for example: >>>> >>>> def worker(): >>>> for item in queue: >>>> with nogil: >>>> do_stuff(item) >>>> >>>> queue.extend(work_items) >>>> start_threads(worker, count) >>>> >>>> Note that the queue is only needed to tell the thread what to work >>>> on. A lot of things can be shared over the closure. So the queue may >>>> not even be required in many cases. >>> >>> I like this approach too. I suppose that you will need to annotate the >>> items so that they are not Python objects, no? Something like: >>> >>> def worker(): >>> cdef int item # tell that item is not a Python object! >>> for item in queue: >>> with nogil: >>> do_stuff(item) >>> >>> queue.extend(work_items) >>> start_threads(worker, count) >> >> On a slightly higher level, are we just trying to use OpenMP from >> Cython, or are we trying to build it into the language? If the former, >> it may make sense to stick closer than one might otherwise be tempted >> in terms of API to the underlying C to leverage the existing >> documentation. A library with a more Pythonic interface could perhaps >> be written on top of that. Alternatively, if we're building it into >> Cython itself, I'd it might be worth modeling it after the >> multiprocessing module (though I understand it would be implemented >> with threads), which I think is a decent enough model for managing >> embarrassingly parallel operations. > > +1 > > >> The above code is similar to that, >> though I'd prefer the for loop implicit rather than as part of the >> worker method (or at least as an argument). > > It provides a simple way to write per-thread initialisation code, > though. And it's likely easier to make looping fast than to speed up > the call into a closure. However, eventually, both ways will need to > be supported anyway. > > >> If we went this route, >> what are the advantages of using OpenMP over, say, pthreads in the >> background? (And could the latter be done with just a library + some >> fancy GIL specifications?) > > In the above example, basically everything is explicit and nothing > more than a simplified threading setup is needed. Even the > implementation of "start_threads()" could be done in a couple of lines > of Python code, including the collection of results and errors. If > someone thinks we need more than that, I'd like to see a couple of > concrete use cases and code examples first. > > >> One thing that's nice about OpenMP as >> implemented in C is that the serial code looks almost exactly like the >> parallel code; the code at http://wiki.cython.org/enhancements/openmp >> has this property too. > > Writing it with a closure isn't really that much different. You can > put the inner function right where it would normally get executed and > add a bit of calling/load distributing code below it. Not that bad IMO. > > It may be worth providing some ready-to-use decorators to do the load > balancing, but I don't really like the idea of having a decorator > magically invoke the function in-place that it decorates. > > >> Also, I like the idea of being able to hold the GIL by the invoking >> thread and having the "sharing" threads do the appropriate locking >> among themselves when needed if possible, e.g. for exception raising. > > I like the explicit "with nogil" block in my example above. It makes > it easy to use normal Python setup code, to synchronise based on the > GIL if desired (e.g. to use a normal Python queue for communication), > and it's simple enough not to get in the way. I'm supporting Robert here. Basically, I'm +1 to anything that can make me pretend the GIL doesn't exist, even if it comes with a 2x performance hit: Because that will make me write parallell code (which I can't be bothered to do in Cython currently), and I have 4 cores on the laptop I use for debugging, so I'd still get a 2x speedup. Perhaps the long-term solution is something like an "autogil" mode could work where Cython automatically releases the GIL on blocks where it can (such as a typed for-loop), and acquires it back when needed (an exception-raising if-block within said for-loop). And when doing multi-threading, GIL-requiring calls are dispatched to a master GIL-holding thread (which would not be a worker thread, i.e. on 4 cores you'd have 4 workers + 1 GIL-holding support thread). So the advice for speeding up code is simply "make sure your code is all typed", just like before, but people can follow that advice without even having to learn about the GIL. It's all about a) lowering learning curve for trivial purposes, b) allow inserting temporary debug print statements using the GIL without having to rework the code. As for the discussion we had on using the GIL for locking, I think that should be made explicit, even if it is a noop currently. I once wrote code relying on the GIL, and really missed something like "cython.gil.lock()" to put in there just for better code readability (yes, I used comments, but...). > > I think it simplifies things a lot when code can rely on the GIL being > held when entering the thread function. Threading is complicated > enough to keep it as explicit as possible. That's exactly the thing about OpenMP: It tends to hide the complexity of threading and allow you to get on with your life. When you say this, it sounds a bit like "people who don't want to learn the technical inner details of Python should just use another language than Cython". If I write code in Fortran it may get parallelized, whereas I almost never write parallel code in Cython (well, MPI, but not shared-memory), all the "is-the-gil-held-or-not" is just too much too keep in my head. Dag Sverre From matej at laitl.cz Fri Mar 11 11:42:26 2011 From: matej at laitl.cz (Matej Laitl) Date: Fri, 11 Mar 2011 11:42:26 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <201103082016.02413.faltet@pytables.org> Message-ID: <1532918.uh6R0CtLRz@esprimo> > On a slightly higher level, are we just trying to use OpenMP from > Cython, or are we trying to build it into the language? If the former, > it may make sense to stick closer than one might otherwise be tempted > in terms of API to the underlying C to leverage the existing > documentation. A library with a more Pythonic interface could perhaps > be written on top of that. Alternatively, if we're building it into > Cython itself, I'd it might be worth modeling it after the > multiprocessing module (though I understand it would be implemented > with threads), which I think is a decent enough model for managing > embarrassingly parallel operations. The above code is similar to that, > though I'd prefer the for loop implicit rather than as part of the > worker method (or at least as an argument). If we went this route, > what are the advantages of using OpenMP over, say, pthreads in the > background? (And could the latter be done with just a library + some > fancy GIL specifications?) One thing that's nice about OpenMP as > implemented in C is that the serial code looks almost exactly like the > parallel code; the code at http://wiki.cython.org/enhancements/openmp > has this property too. +1. I'm strongly for implementing thin and low-level support for OpenMP at the first place instead of (ab?)using it to implement high-level threading API. Also, code like that would have an advantage (significant for my project[1]) of being compilable by older cython / interpretable by python with no cython at all ("pure" pure-python mode): #pragma omp parallel for private(var1) reduction(+:var2) schedule(guided) for i in range(n): do_work(i) [1] http://github.com/strohel/PyBayes Also, the implementation should be straightforward, I use to patch generated .c manually and only python ? c variable name translation is needed, + perhaps taking care of temp variables that need to be thread-local. My five cents, Mat?j Laitl From sturla at molden.no Fri Mar 11 12:13:28 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 11 Mar 2011 12:13:28 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> Message-ID: <4D7A03D8.3020705@molden.no> Den 11.03.2011 01:46, skrev Robert Bradshaw: > On a slightly higher level, are we just trying to use OpenMP from > Cython, or are we trying to build it into the language? OpenMP is a specification, not a particular implementation. Implementation for Cython should either be compiler pragmas or a library. I'd like it to be a library, as it should also be usable from Python. I have made some progress on the library route, depending on Cython's closures. nogil makes things feel a bit awkward, though. We could for example imagine code like this: with openmp.ordered(i): Context managers are forbidden in nogil AFAIK. So we end up with ugly hacks like this: with nogil: if openmp._ordered(i): # always returns 1, but will synchronize Would it be possible to: - Make context managers that are allowed without the GIL? We don't need to worry about exceptions, but it should be possible to short-circuit from __enter__ to __exit__. - Have cpdefs that are callable without the GIL? This would certainly make OpenMP syntax look cleaner. Sturla From stefan_ml at behnel.de Fri Mar 11 12:37:50 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Mar 2011 12:37:50 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D79D5C3.7080805@student.matnat.uio.no> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D79CD3C.6050004@behnel.de> <4D79D5C3.7080805@student.matnat.uio.no> Message-ID: <4D7A098E.7090305@behnel.de> Dag Sverre Seljebotn, 11.03.2011 08:56: > Basically, I'm +1 to anything that can make me > pretend the GIL doesn't exist, even if it comes with a 2x performance hit: > Because that will make me write parallell code (which I can't be bothered > to do in Cython currently), and I have 4 cores on the laptop I use for > debugging, so I'd still get a 2x speedup. > > Perhaps the long-term solution is something like an "autogil" mode could > work where Cython automatically releases the GIL on blocks where it can > (such as a typed for-loop), and acquires it back when needed (an > exception-raising if-block within said for-loop). I assume you mean this to become a decorator or other option written into the code. > And when doing > multi-threading, GIL-requiring calls are dispatched to a master GIL-holding > thread (which would not be a worker thread, i.e. on 4 cores you'd have 4 > workers + 1 GIL-holding support thread). So the advice for speeding up code > is simply "make sure your code is all typed", just like before, but people > can follow that advice without even having to learn about the GIL. The GIL does not only protect the interpreter core. It also protects C level data structures in user code and keeps threaded code from running amok. Releasing and acquiring it doesn't come for free either, so besides likely breaking code that was not specifically written to be reentrant, releasing it automatically may also introduce a performance penalty for many users. I'm very happy the GIL exists, and I'm against anything that tries to disable it automatically. Threading is an extremely dangerous programming model. The GIL has its gotchas, too, but it still simplifies it quite a bit. Actually, threading is so complex and easy to get wrong, that any threaded code should always be written specifically to support threading. Explicitly acquiring and releasing the GIL is really just a minor issue on that path. Stefan From stefan_ml at behnel.de Fri Mar 11 12:43:30 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Mar 2011 12:43:30 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7A03D8.3020705@molden.no> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D7A03D8.3020705@molden.no> Message-ID: <4D7A0AE2.2010704@behnel.de> Sturla Molden, 11.03.2011 12:13: > OpenMP is a specification, not a particular implementation. Implementation > for Cython should either be compiler pragmas or a library. > > I'd like it to be a library, as it should also be usable from Python. I > have made some progress on the library route, depending on Cython's closures. > > nogil makes things feel a bit awkward, though. > > We could for example imagine code like this: > > with openmp.ordered(i): > > > Context managers are forbidden in nogil AFAIK. So we end up with ugly hacks > like this: > > with nogil: > if openmp._ordered(i): # always returns 1, but will synchronize > > > Would it be possible to: > > - Make context managers that are allowed without the GIL? We don't need to > worry about exceptions, but it should be possible to short-circuit from > __enter__ to __exit__. The two special methods are Python methods. There is no way to call them without holding the GIL. If we wanted to enable something like a context manager inside of a nogil block, it would necessarily be a different protocol. > - Have cpdefs that are callable without the GIL? "cpdef" functions are meant to be overridable from Python code, so, no, you cannot call them from within a nogil block as they may actually have been overridden. What's your use actual case for this? Stefan From d.s.seljebotn at astro.uio.no Fri Mar 11 13:15:46 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 11 Mar 2011 13:15:46 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7A098E.7090305@behnel.de> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D79CD3C.6050004@behnel.de> <4D79D5C3.7080805@student.matnat.uio.no> <4D7A098E.7090305@behnel.de> Message-ID: <4D7A1272.2030108@student.matnat.uio.no> On 03/11/2011 12:37 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 11.03.2011 08:56: >> Basically, I'm +1 to anything that can make me >> pretend the GIL doesn't exist, even if it comes with a 2x performance >> hit: >> Because that will make me write parallell code (which I can't be >> bothered >> to do in Cython currently), and I have 4 cores on the laptop I use for >> debugging, so I'd still get a 2x speedup. >> >> Perhaps the long-term solution is something like an "autogil" mode could >> work where Cython automatically releases the GIL on blocks where it can >> (such as a typed for-loop), and acquires it back when needed (an >> exception-raising if-block within said for-loop). > > I assume you mean this to become a decorator or other option written > into the code. > > >> And when doing >> multi-threading, GIL-requiring calls are dispatched to a master >> GIL-holding >> thread (which would not be a worker thread, i.e. on 4 cores you'd have 4 >> workers + 1 GIL-holding support thread). So the advice for speeding >> up code >> is simply "make sure your code is all typed", just like before, but >> people >> can follow that advice without even having to learn about the GIL. > > The GIL does not only protect the interpreter core. It also protects C > level data structures in user code and keeps threaded code from > running amok. Releasing and acquiring it doesn't come for free either, > so besides likely breaking code that was not specifically written to > be reentrant, releasing it automatically may also introduce a > performance penalty for many users. The intention was that the GIL would be acquired in exceptional circumstances (doesn't matter for overall performance) or during debugging (again don't care about performance). But I agree the idea needs more thought on the possible pitfalls. > > I'm very happy the GIL exists, and I'm against anything that tries to > disable it automatically. Threading is an extremely dangerous > programming model. The GIL has its gotchas, too, but it still > simplifies it quite a bit. Actually, threading is so complex and easy > to get wrong, that any threaded code should always be written > specifically to support threading. Explicitly acquiring and releasing > the GIL is really just a minor issue on that path. I guess the point is that OpenMP takes that "extremely dangerous programming model" and makes it tractable, at least for a class of trivial problems (not necessarily SIMD, but almost). BTW, threading is often used simply because how how array data is laid out in memory. Typical usecase is every thread write to different non-overlapping blocks of the same array (and read from the same input arrays that are not changed). Then you move on to step B, which does the same, but perhaps blocks the arrays in a different way between threads. Then step C blocks the data in yet another way, etc. But at each step it's just "input arrays, non-overlapping blocks in output arrays", global parameters, local loop counters. (One doesn't need to use threads, there was another thread on multiprocessing + shared memory arrays.) Just saying that not all use of threads is "extremely dangerous", and OpenMP exists explicitly to dumb threading down for those cases. Dag Sverre From stefan_ml at behnel.de Fri Mar 11 13:45:11 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Mar 2011 13:45:11 +0100 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) Message-ID: <4D7A1957.6070107@behnel.de> Hi, ticket 654 describes a code generation problem where the arguments to a cdef function are not being evaluated in the order they are written down in the code. http://trac.cython.org/cython_trac/ticket/654 This introduces problems when the arguments have side effects or are not simple, e.g. they are function calls themselves or are taken from object attributes or live in a closure. For example, f(g(a), a.x, h(a)) will produce different results depending on the evaluation order if g or h change the value of a.x. However, apparently, the order of evaluation is only guaranteed by Python, not by C. Now, the question is: what are the right semantics for Cython here: follow Python or C? Personally, I think it would be nice to keep up Python's semantics, but when I implemented this I broke quite some code in Sage (you may have noticed that the sage-build project in Hudson has been red for a while). There are things in C and especially in C++ that cannot be easily copied into a temporary variable in order to make sure they are evaluated before the following arguments. This is not a problem for Python function calls where all arguments end up being copied (and often converted) anyway. It is a problem for C function calls, though. What do you think about this? Stefan From sturla at molden.no Fri Mar 11 14:54:45 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 11 Mar 2011 14:54:45 +0100 Subject: [Cython] OpenMP support In-Reply-To: <1532918.uh6R0CtLRz@esprimo> References: <201103082016.02413.faltet@pytables.org> <1532918.uh6R0CtLRz@esprimo> Message-ID: <4D7A29A5.2060503@molden.no> Den 11.03.2011 11:42, skrev Matej Laitl: > #pragma omp parallel for private(var1) reduction(+:var2) schedule(guided) > for i in range(n): > do_work(i) > I do like this, as it is valid Python and can be turned on/off with a compiler flag to Cython. Issues to warn about: - We cannot jump out of a parallel block from C/C++/Fortran (goto, longjmp, C++ exception). That applies to Python exceptions as well, and the generated Cython code. - GIL issue: CPython interpreter actually call GetCurrentThreadId(), e.g. in thread_nt.h. So the OpenMP thread using the Python CAPI must be the OS thread holding the GIL. It is not sufficient that the master thread does. - Remember that NumPy arrays are unboxed. Those local variables should be silently passed as firstprivate. - Refcounting with Python objects and private variables. None of the above applies if we go with the library approach. But then it would look less like OpenMP in C. Also, do we want this? #pragma omp parallel if 1: It is a consequence of just re-using the C-syntax for OpenMP, as intendation matters in Cython. There are no anonymous blocks similar to C in Cython: #pragma omp parallel { } Sturla From vitja.makarov at gmail.com Fri Mar 11 15:04:24 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 11 Mar 2011 17:04:24 +0300 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D7A1957.6070107@behnel.de> References: <4D7A1957.6070107@behnel.de> Message-ID: 2011/3/11 Stefan Behnel : > Hi, > > ticket 654 describes a code generation problem where the arguments to a cdef > function are not being evaluated in the order they are written down in the > code. > > http://trac.cython.org/cython_trac/ticket/654 > > This introduces problems when the arguments have side effects or are not > simple, e.g. they are function calls themselves or are taken from object > attributes or live in a closure. For example, > > ? ?f(g(a), a.x, h(a)) > > will produce different results depending on the evaluation order if g or h > change the value of a.x. > > However, apparently, the order of evaluation is only guaranteed by Python, > not by C. Now, the question is: what are the right semantics for Cython > here: follow Python or C? > +1 for Python, Cython is mostly Python than C, btw I don't think that it makes much sense. > Personally, I think it would be nice to keep up Python's semantics, but when > I implemented this I broke quite some code in Sage (you may have noticed > that the sage-build project in Hudson has been red for a while). There are > things in C and especially in C++ that cannot be easily copied into a > temporary variable in order to make sure they are evaluated before the > following arguments. This is not a problem for Python function calls where > all arguments end up being copied (and often converted) anyway. It is a > problem for C function calls, though. > > What do you think about this? > > ? ?f(g(a), a.x, h(a)) Why could not this be translated into: tmp1 = g(a) tmp2 = a.x tmp3 = h(a) f(tmp1, tmp2, tmp3) -- vitja. From stefan_ml at behnel.de Fri Mar 11 15:08:13 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Mar 2011 15:08:13 +0100 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: References: <4D7A1957.6070107@behnel.de> Message-ID: <4D7A2CCD.6060604@behnel.de> Vitja Makarov, 11.03.2011 15:04: > 2011/3/11 Stefan Behnel: >> Personally, I think it would be nice to keep up Python's semantics, but when >> I implemented this I broke quite some code in Sage (you may have noticed >> that the sage-build project in Hudson has been red for a while). There are >> things in C and especially in C++ that cannot be easily copied into a >> temporary variable in order to make sure they are evaluated before the >> following arguments. This is not a problem for Python function calls where >> all arguments end up being copied (and often converted) anyway. It is a >> problem for C function calls, though. > >> f(g(a), a.x, h(a)) > > Why could not this be translated into: > > tmp1 = g(a) > tmp2 = a.x > tmp3 = h(a) > > f(tmp1, tmp2, tmp3) See above. Stefan From sturla at molden.no Fri Mar 11 15:19:23 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 11 Mar 2011 15:19:23 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7A0AE2.2010704@behnel.de> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D7A03D8.3020705@molden.no> <4D7A0AE2.2010704@behnel.de> Message-ID: <4D7A2F6B.6060407@molden.no> Den 11.03.2011 12:43, skrev Stefan Behnel: > > What's your use actual case for this? > Just avoid different syntax inside and outside nogil-blocks. I like this style with openmp.critical: better than what is currently legal with nogil: openmp.critical() if 1: openmp.end_critical() Sturla From stefan_ml at behnel.de Fri Mar 11 15:24:06 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Mar 2011 15:24:06 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7A2F6B.6060407@molden.no> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D7A03D8.3020705@molden.no> <4D7A0AE2.2010704@behnel.de> <4D7A2F6B.6060407@molden.no> Message-ID: <4D7A3086.6010308@behnel.de> Sturla Molden, 11.03.2011 15:19: > Den 11.03.2011 12:43, skrev Stefan Behnel: >> >> What's your use actual case for this? >> > > Just avoid different syntax inside and outside nogil-blocks. I like this style > > with openmp.critical: > > > better than what is currently legal with nogil: > > openmp.critical() > if 1: > > openmp.end_critical() No, I meant, what is your specific need to "cpdef" in this context? Stefan From dalcinl at gmail.com Fri Mar 11 16:05:25 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 11 Mar 2011 12:05:25 -0300 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D7A1957.6070107@behnel.de> References: <4D7A1957.6070107@behnel.de> Message-ID: On 11 March 2011 09:45, Stefan Behnel wrote: > Hi, > > ticket 654 describes a code generation problem where the arguments to a cdef > function are not being evaluated in the order they are written down in the > code. > > http://trac.cython.org/cython_trac/ticket/654 > > This introduces problems when the arguments have side effects or are not > simple, e.g. they are function calls themselves or are taken from object > attributes or live in a closure. For example, > > ? ?f(g(a), a.x, h(a)) > > will produce different results depending on the evaluation order if g or h > change the value of a.x. > > However, apparently, the order of evaluation is only guaranteed by Python, > not by C. Now, the question is: what are the right semantics for Cython > here: follow Python or C? > > Personally, I think it would be nice to keep up Python's semantics, but when > I implemented this I broke quite some code in Sage (you may have noticed > that the sage-build project in Hudson has been red for a while). There are > things in C and especially in C++ that cannot be easily copied into a > temporary variable in order to make sure they are evaluated before the > following arguments. This is not a problem for Python function calls where > all arguments end up being copied (and often converted) anyway. It is a > problem for C function calls, though. > > What do you think about this? > Regarding our previous history, I would go for Python semantics... Perhaps you could add a compiler directive for these rare cases where you need/want C/C++ semantics? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Mar 11 18:36:47 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 11 Mar 2011 18:36:47 +0100 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D7A2CCD.6060604@behnel.de> References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> Message-ID: <4D7A5DAF.1020003@behnel.de> Stefan Behnel, 11.03.2011 15:08: > Vitja Makarov, 11.03.2011 15:04: >> 2011/3/11 Stefan Behnel: >>> Personally, I think it would be nice to keep up Python's semantics, but >>> when >>> I implemented this I broke quite some code in Sage (you may have noticed >>> that the sage-build project in Hudson has been red for a while). There are >>> things in C and especially in C++ that cannot be easily copied into a >>> temporary variable in order to make sure they are evaluated before the >>> following arguments. This is not a problem for Python function calls where >>> all arguments end up being copied (and often converted) anyway. It is a >>> problem for C function calls, though. >> >>> f(g(a), a.x, h(a)) >> >> Why could not this be translated into: >> >> tmp1 = g(a) >> tmp2 = a.x >> tmp3 = h(a) >> >> f(tmp1, tmp2, tmp3) > > See above. To be a little clearer here, it's a problem in C for example with struct values. Copying them by value into a temp variable can be expensive, potentially twice as expensive as simply passing them into the function normally. Not sure what kind of additional devilry C++ provides here, but I'd expect that object values can exhibit bizarre behaviour when being assigned. Maybe others can enlighten me here. I have no idea how many cases there actually are that we can't handle or that may lead to a performance degradation when using temps, but the problem is that they exist at all. Stefan From faltet at pytables.org Fri Mar 11 19:03:13 2011 From: faltet at pytables.org (Francesc Alted) Date: Fri, 11 Mar 2011 19:03:13 +0100 Subject: [Cython] OpenMP support In-Reply-To: <1532918.uh6R0CtLRz@esprimo> References: <1532918.uh6R0CtLRz@esprimo> Message-ID: <201103111903.13552.faltet@pytables.org> A Friday 11 March 2011 11:42:26 Matej Laitl escrigu?: > I'm strongly for implementing thin and low-level support for OpenMP > at the first place instead of (ab?)using it to implement high-level > threading API. My opinion on this continues to be -1. I'm afraid that difficult access to OpenMP on Windows platforms (lack of OpenMP in MSVC Express is, as I see it, a major showstopper, although perhaps GCC 4.x on Win is stable enough already, I don't know) would prevent of *true* portability of OpenMP-powered Cython programs. IMHO, going to the native Python threads + possibly new Cython syntax is a better venue. But I'd like to be proved that the problem for Win is not that grave... -- Francesc Alted From robertwb at math.washington.edu Fri Mar 11 19:33:02 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 11 Mar 2011 10:33:02 -0800 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D7A5DAF.1020003@behnel.de> References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> Message-ID: On Fri, Mar 11, 2011 at 9:36 AM, Stefan Behnel wrote: > Stefan Behnel, 11.03.2011 15:08: >> >> Vitja Makarov, 11.03.2011 15:04: >>> >>> 2011/3/11 Stefan Behnel: >>>> >>>> Personally, I think it would be nice to keep up Python's semantics, but >>>> when >>>> I implemented this I broke quite some code in Sage (you may have noticed >>>> that the sage-build project in Hudson has been red for a while). There >>>> are >>>> things in C and especially in C++ that cannot be easily copied into a >>>> temporary variable in order to make sure they are evaluated before the >>>> following arguments. This is not a problem for Python function calls >>>> where >>>> all arguments end up being copied (and often converted) anyway. It is a >>>> problem for C function calls, though. >>> >>>> f(g(a), a.x, h(a)) >>> >>> Why could not this be translated into: >>> >>> tmp1 = g(a) >>> tmp2 = a.x >>> tmp3 = h(a) >>> >>> f(tmp1, tmp2, tmp3) >> >> See above. > > To be a little clearer here, it's a problem in C for example with struct > values. Copying them by value into a temp variable can be expensive, > potentially twice as expensive as simply passing them into the function > normally. Yep, and some types (e.g. array types) can't be assigned to at all. FWIW, the issues with Sage is that many libraries use the "stack allocated, pass-by-reference" trick typedef foo_struct foo_t[1] but we have declared them to be of type "void*" because we don't care about or want to muck with the internals. Cleaning this up is something we should do, but is taking a while, and aside from that it makes Cython even more dependent on correct type declarations (and backwards incompatible in this regard). Sage is a great test for the buildbot, so keeping it red for so long is not a good thing either. > Not sure what kind of additional devilry C++ provides here, but I'd expect > that object values can exhibit bizarre behaviour when being assigned. Maybe > others can enlighten me here. Yes, C++ allows overloading of the assignment operator, so assigning may lead to arbitrary code (as well as probably an expensive copy, as with structs, and structs in C++ are really just classes with a different visibility). > I have no idea how many cases there actually are that we can't handle or > that may lead to a performance degradation when using temps, but the problem > is that they exist at all. Note that this applies not just to function arguments, but a host of other places, e.g. the order of evaluation of "f() + g()" in C is unspecified. Fleshing this out completely will lead to a lot more temps and verbose C code. And then we'd just cross our fingers that the C compiler was able to optimize all these temps away (though still possibly producing inferior code if it were able to switch order of execution). Whatever we do, we need a flag/directive. Perhaps there's a way to guarantee correct ordering for all valid Python code, even in the presence of (function and variable) type inference, but allow some leeway for explicitly declared C functions. - Robert From greg.ewing at canterbury.ac.nz Sat Mar 12 00:29:28 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 12 Mar 2011 12:29:28 +1300 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D7A1957.6070107@behnel.de> References: <4D7A1957.6070107@behnel.de> Message-ID: <4D7AB058.8000600@canterbury.ac.nz> Stefan Behnel wrote: > This introduces problems when the arguments have side effects or are not > simple, e.g. > > f(g(a), a.x, h(a)) > > What do you think about this? I think it's a bad idea to write code that relies on the order of evaluation like this. If the order matters, it's better to be explicit about it: arg1 = g(a) arg2 = h(a) f(arg1, a.x, arg2) So I would say it's not something worth worrying about overly much. -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 12 00:37:11 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 12 Mar 2011 12:37:11 +1300 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D7A5DAF.1020003@behnel.de> References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> Message-ID: <4D7AB227.3060303@canterbury.ac.nz> Stefan Behnel wrote: > To be a little clearer here, it's a problem in C for example with struct > values. Copying them by value into a temp variable can be expensive, > potentially twice as expensive as simply passing them into the function > normally. What are you actually proposing to do here? Anything that Cython does in the generated code to guarantee evaluation order is going to involve using temp variables, and thus incur the same overhead. -- Greg From sturla at molden.no Sat Mar 12 02:43:16 2011 From: sturla at molden.no (Sturla Molden) Date: Sat, 12 Mar 2011 02:43:16 +0100 Subject: [Cython] OpenMP support In-Reply-To: <201103111903.13552.faltet@pytables.org> References: <1532918.uh6R0CtLRz@esprimo> <201103111903.13552.faltet@pytables.org> Message-ID: <74694C47-588C-4F31-9680-CDDB0D65EA67@molden.no> The free C/C++ compiler in Windows SDK supports OpenMP. This is the system C compiler on Windows. Visual C++ Express is an IDE for beginners and hobbyists. OpenMP on GCC is the same on Windows as on any other platform. Sturla > A Friday 11 March 2011 11:42:26 Matej Laitl escrigu?: >> I'm strongly for implementing thin and low-level support for OpenMP >> at the first place instead of (ab?)using it to implement high-level >> threading API. > > My opinion on this continues to be -1. I'm afraid that difficult > access > to OpenMP on Windows platforms (lack of OpenMP in MSVC Express is, > as I > see it, a major showstopper, although perhaps GCC 4.x on Win is stable > enough already, I don't know) would prevent of *true* portability of > OpenMP-powered Cython programs. > > IMHO, going to the native Python threads + possibly new Cython > syntax is > a better venue. But I'd like to be proved that the problem for Win is > not that grave... > > -- > Francesc Alted > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From faltet at pytables.org Sat Mar 12 11:58:32 2011 From: faltet at pytables.org (Francesc Alted) Date: Sat, 12 Mar 2011 11:58:32 +0100 Subject: [Cython] OpenMP support In-Reply-To: <74694C47-588C-4F31-9680-CDDB0D65EA67@molden.no> References: <201103111903.13552.faltet@pytables.org> <74694C47-588C-4F31-9680-CDDB0D65EA67@molden.no> Message-ID: <201103121158.32437.faltet@pytables.org> A Saturday 12 March 2011 02:43:16 Sturla Molden escrigu?: > The free C/C++ compiler in Windows SDK supports OpenMP. This is the > system C compiler on Windows. System compiler on Windows? I've never heard defining MSVC like this ;) > OpenMP on GCC is the same on Windows as on any other platform. In principle yes. But it is curious how the NumPy crew are still not being able to offer 64-bit binaries for Windows. Could not they use the 'system C compiler'? Could not they use the GCC 4.x for Win? My impression is that this compiler issue on Win is more hairy than you are suggesting. Not that I'm a big fan of Windows platforms, but we have to accept the fact that its user base is simply huge. -- Francesc Alted From markflorisson88 at gmail.com Sat Mar 12 13:11:26 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 12 Mar 2011 13:11:26 +0100 Subject: [Cython] OpenMP support In-Reply-To: References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> Message-ID: On 11 March 2011 01:46, Robert Bradshaw wrote: > On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted wrote: >> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigu?: >>> mark florisson, 08.03.2011 18:00: >>> > What I meant was that the >>> > wrapper returned by the decorator would have to call the closure >>> > for every iteration, which introduces function call overhead. >>> > >>> >[...] >>> > >>> > I guess we just have to establish what we want to do: do we >>> > want to support code with Python objects (and exceptions etc), or >>> > just C code written in Cython? >>> >>> I like the approach that Sturla mentioned: using closures to >>> implement worker threads. I think that's very pythonic. You could do >>> something like this, for example: >>> >>> ? ? ?def worker(): >>> ? ? ? ? ?for item in queue: >>> ? ? ? ? ? ? ?with nogil: >>> ? ? ? ? ? ? ? ? ?do_stuff(item) >>> >>> ? ? ?queue.extend(work_items) >>> ? ? ?start_threads(worker, count) >>> >>> Note that the queue is only needed to tell the thread what to work >>> on. A lot of things can be shared over the closure. So the queue may >>> not even be required in many cases. >> >> I like this approach too. ?I suppose that you will need to annotate the >> items so that they are not Python objects, no? ?Something like: >> >> ? ? def worker(): >> ? ? ? ? cdef int item ?# tell that item is not a Python object! >> ? ? ? ? for item in queue: >> ? ? ? ? ? ? with nogil: >> ? ? ? ? ? ? ? ? do_stuff(item) >> >> ? ? queue.extend(work_items) >> ? ? start_threads(worker, count) > > On a slightly higher level, are we just trying to use OpenMP from > Cython, or are we trying to build it into the language? If the former, > it may make sense to stick closer than one might otherwise be tempted > in terms of API to the underlying C to leverage the existing > documentation. A library with a more Pythonic interface could perhaps > be written on top of that. Alternatively, if we're building it into > Cython itself, I'd it might be worth modeling it after the > multiprocessing module (though I understand it would be implemented > with threads), which I think is a decent enough model for managing > embarrassingly parallel operations. The above code is similar to that, > though I'd prefer the for loop implicit rather than as part of the > worker method (or at least as an argument). If we went this route, > what are the advantages of using OpenMP over, say, pthreads in the > background? (And could the latter be done with just a library + some > fancy GIL specifications?) One thing that's nice about OpenMP as > implemented in C is that the serial code looks almost exactly like the > parallel code; the code at http://wiki.cython.org/enhancements/openmp > has this property too. Indeed, and this works very nicely for for loops because they already constitute blocks. However, if you want to support other OpenMP-like functionality (and not saying that we want to, but if), then you have two options: 1) you need a way to explicitly declare blocks in Cython, or 2) you need to write all your blocks as functions and call those, instead (most likely quite bothersome). If we want an interface like multiprocessing, I think we can just existing batteries, of which just one example is the functionality described in PEP 3148 (concurrent.futures, new in 3.2). People would only have to write Python functions that could release the GIL if needed. > Also, I like the idea of being able to hold the GIL by the invoking > thread and having the "sharing" threads do the appropriate locking > among themselves when needed if possible, e.g. for exception raising. > > Another thought I had is, there might be other usecases for being able > to emit generic pragmas statements, how far would that get us? I gave that, but more generally, a 'verbatim' clause some thought. However, syntax would be bothersome. I thought about e.g. cdef verbatim: C code here Access to variables (i.e., substitute Cython variable names with their name-mangled counterparts) might be provided through e.g. '$var' syntax, as used by string.Template. But I can already hear people object rather loudly, so I rejected the thought :). If, of course, access to variables is not allowed but instead we allow only simple pragmas, a simple cython.pragma("#pragma bla") should probably suffice. On the other hand, do we really need to mangle (non-temporary) variable names? Global variables are declared static and variables may be redeclared in inner C scopes. > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Sat Mar 12 13:12:08 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 12 Mar 2011 13:12:08 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D79D5C3.7080805@student.matnat.uio.no> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D79CD3C.6050004@behnel.de> <4D79D5C3.7080805@student.matnat.uio.no> Message-ID: On 11 March 2011 08:56, Dag Sverre Seljebotn wrote: > On 03/11/2011 08:20 AM, Stefan Behnel wrote: >> >> Robert Bradshaw, 11.03.2011 01:46: >>> >>> On Tue, Mar 8, 2011 at 11:16 AM, Francesc Alted >>> ?wrote: >>>> >>>> A Tuesday 08 March 2011 18:50:15 Stefan Behnel escrigu?: >>>>> >>>>> mark florisson, 08.03.2011 18:00: >>>>>> >>>>>> What I meant was that the >>>>>> wrapper returned by the decorator would have to call the closure >>>>>> for every iteration, which introduces function call overhead. >>>>>> >>>>>> [...] >>>>>> >>>>>> I guess we just have to establish what we want to do: do we >>>>>> want to support code with Python objects (and exceptions etc), or >>>>>> just C code written in Cython? >>>>> >>>>> I like the approach that Sturla mentioned: using closures to >>>>> implement worker threads. I think that's very pythonic. You could do >>>>> something like this, for example: >>>>> >>>>> ? ? ?def worker(): >>>>> ? ? ? ? ?for item in queue: >>>>> ? ? ? ? ? ? ?with nogil: >>>>> ? ? ? ? ? ? ? ? ?do_stuff(item) >>>>> >>>>> ? ? ?queue.extend(work_items) >>>>> ? ? ?start_threads(worker, count) >>>>> >>>>> Note that the queue is only needed to tell the thread what to work >>>>> on. A lot of things can be shared over the closure. So the queue may >>>>> not even be required in many cases. >>>> >>>> I like this approach too. ?I suppose that you will need to annotate the >>>> items so that they are not Python objects, no? ?Something like: >>>> >>>> ? ? def worker(): >>>> ? ? ? ? cdef int item ?# tell that item is not a Python object! >>>> ? ? ? ? for item in queue: >>>> ? ? ? ? ? ? with nogil: >>>> ? ? ? ? ? ? ? ? do_stuff(item) >>>> >>>> ? ? queue.extend(work_items) >>>> ? ? start_threads(worker, count) >>> >>> On a slightly higher level, are we just trying to use OpenMP from >>> Cython, or are we trying to build it into the language? If the former, >>> it may make sense to stick closer than one might otherwise be tempted >>> in terms of API to the underlying C to leverage the existing >>> documentation. A library with a more Pythonic interface could perhaps >>> be written on top of that. Alternatively, if we're building it into >>> Cython itself, I'd it might be worth modeling it after the >>> multiprocessing module (though I understand it would be implemented >>> with threads), which I think is a decent enough model for managing >>> embarrassingly parallel operations. >> >> +1 >> >> >>> The above code is similar to that, >>> though I'd prefer the for loop implicit rather than as part of the >>> worker method (or at least as an argument). >> >> It provides a simple way to write per-thread initialisation code, though. >> And it's likely easier to make looping fast than to speed up the call into a >> closure. However, eventually, both ways will need to be supported anyway. >> >> >>> If we went this route, >>> what are the advantages of using OpenMP over, say, pthreads in the >>> background? (And could the latter be done with just a library + some >>> fancy GIL specifications?) >> >> In the above example, basically everything is explicit and nothing more >> than a simplified threading setup is needed. Even the implementation of >> "start_threads()" could be done in a couple of lines of Python code, >> including the collection of results and errors. If someone thinks we need >> more than that, I'd like to see a couple of concrete use cases and code >> examples first. >> >> >>> One thing that's nice about OpenMP as >>> implemented in C is that the serial code looks almost exactly like the >>> parallel code; the code at http://wiki.cython.org/enhancements/openmp >>> has this property too. >> >> Writing it with a closure isn't really that much different. You can put >> the inner function right where it would normally get executed and add a bit >> of calling/load distributing code below it. Not that bad IMO. >> >> It may be worth providing some ready-to-use decorators to do the load >> balancing, but I don't really like the idea of having a decorator magically >> invoke the function in-place that it decorates. >> >> >>> Also, I like the idea of being able to hold the GIL by the invoking >>> thread and having the "sharing" threads do the appropriate locking >>> among themselves when needed if possible, e.g. for exception raising. >> >> I like the explicit "with nogil" block in my example above. It makes it >> easy to use normal Python setup code, to synchronise based on the GIL if >> desired (e.g. to use a normal Python queue for communication), and it's >> simple enough not to get in the way. > > I'm supporting Robert here. Basically, I'm +1 to anything that can make me > pretend the GIL doesn't exist, even if it comes with a 2x performance hit: > Because that will make me write parallell code (which I can't be bothered to > do in Cython currently), and I have 4 cores on the laptop I use for > debugging, so I'd still get a 2x speedup. > > Perhaps the long-term solution is something like an "autogil" mode could > work where Cython automatically releases the GIL on blocks where it can > (such as a typed for-loop), and acquires it back when needed (an > exception-raising if-block within said for-loop). And when doing > multi-threading, GIL-requiring calls are dispatched to a master GIL-holding > thread (which would not be a worker thread, i.e. on 4 cores you'd have 4 > workers + 1 GIL-holding support thread). So the advice for speeding up code > is simply "make sure your code is all typed", just like before, but people > can follow that advice without even having to learn about the GIL. > > It's all about a) lowering learning curve for trivial purposes, b) allow > inserting temporary debug print statements using the GIL without having to > rework the code. Have we ever thought about supporting 'with gil' as an actual statement instead of just as part of a function declaration or definition? Then you could just say in a 'with nogil:' block: 'with gil: print myvar'. On the other hand, we could also convert usages of 'print' (that is, of simple 'print a, b, c'-style printing) in 'nogil' blocks or functions to C printf statements, where possible. Would that be a wanted feature? > As for the discussion we had on using the GIL for locking, I think that > should be made explicit, even if it is a noop currently. I once wrote code > relying on the GIL, and really missed something like "cython.gil.lock()" to > put in there just for better code readability (yes, I used comments, > but...). > >> >> I think it simplifies things a lot when code can rely on the GIL being >> held when entering the thread function. Threading is complicated enough to >> keep it as explicit as possible. > > That's exactly the thing about OpenMP: It tends to hide the complexity of > threading and allow you to get on with your life. When you say this, it > sounds a bit like "people who don't want to learn the technical inner > details of Python should just use another language than Cython". > > If I write code in Fortran it may get parallelized, whereas I almost never > write parallel code in Cython (well, MPI, but not shared-memory), all the > "is-the-gil-held-or-not" is just too much too keep in my head. > > Dag Sverre > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Sat Mar 12 13:12:26 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 12 Mar 2011 13:12:26 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7A03D8.3020705@molden.no> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D7A03D8.3020705@molden.no> Message-ID: On 11 March 2011 12:13, Sturla Molden wrote: > Den 11.03.2011 01:46, skrev Robert Bradshaw: >> >> On a slightly higher level, are we just trying to use OpenMP from >> Cython, or are we trying to build it into the language? > > OpenMP is a specification, not a particular implementation. Implementation > for Cython should either be compiler pragmas or a library. > > I'd like it to be a library, as it should also be usable from Python. I have > made some progress on the library route, depending on Cython's closures. > > nogil makes things feel a bit awkward, though. > > We could for example imagine code like this: > > ? ?with openmp.ordered(i): > > > Context managers are forbidden in nogil AFAIK. ?So we end up with ugly hacks > like this: > > ? ?with nogil: > ? ? ? if openmp._ordered(i): # always returns 1, but will synchronize > > > Would it be possible to: > > - Make context managers that are allowed without the GIL? We don't need to > worry about exceptions, but it should be possible to short-circuit from > __enter__ to __exit__. > > - Have cpdefs that are callable without the GIL? > > This would certainly make OpenMP syntax look cleaner. > > > Sturla > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > The reason I wanted to support OpenMP as a compiler modification is exactly that you can use Pythonic syntax without actually dealing with for instance, context managers. You'd just rewrite the WithStatNode in a Transform. By additionally stubbing the context managers in the Shadow module you get normal, sequential behaviour when running directly in the python interpreter (provided that Cython is installed or the Shadow module is shipped as cython.py). From markflorisson88 at gmail.com Sat Mar 12 13:12:44 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 12 Mar 2011 13:12:44 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7A29A5.2060503@molden.no> References: <201103082016.02413.faltet@pytables.org> <1532918.uh6R0CtLRz@esprimo> <4D7A29A5.2060503@molden.no> Message-ID: On 11 March 2011 14:54, Sturla Molden wrote: > Den 11.03.2011 11:42, skrev Matej Laitl: >> >> #pragma omp parallel for private(var1) reduction(+:var2) schedule(guided) >> for i in range(n): >> ? ? do_work(i) >> > I do like this, as it is valid Python and can be turned on/off with a > compiler flag to Cython. That's the same thing I proposed with different syntax :) The problem with the pragmas is that we can't declare arbitrary blocks in Cython, to which the pragmas should be applied, in which case get the kind of awkwardness seen below. What I liked about your proposed closures is the neat pythonic syntax for the scoping rules of variables in the loops. I think I might have changed my mind entirely about OpenMP(-like) support, as users can already do a lot, and if they really want OpenMP then it will be likely most convenient to write it in C and simply wrap it. > Issues to warn about: > > - We cannot jump out of a parallel block from C/C++/Fortran (goto, longjmp, > C++ exception). That applies to Python exceptions as well, and the generated > Cython code. > > - GIL issue: CPython interpreter actually call GetCurrentThreadId(), e.g. in > thread_nt.h. So the OpenMP thread using the Python CAPI must be the OS > thread holding the GIL. It is not sufficient that the master thread does. > > - Remember that NumPy arrays are unboxed. Those local variables should be > silently passed as firstprivate. > > - Refcounting with Python objects and private variables. > > None of the above applies if we go with the library approach. But then it > would look less like OpenMP in C. > > Also, do we want this? > > ? #pragma omp parallel > ? if 1: > > > It is a consequence of just re-using the C-syntax for OpenMP, as intendation > matters in Cython. There are no anonymous blocks similar to C in Cython: > > ? #pragma omp parallel > ? { > > ? } > > > > > Sturla > > > > > > > > > > > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From greg.ewing at canterbury.ac.nz Sat Mar 12 23:57:58 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 13 Mar 2011 11:57:58 +1300 Subject: [Cython] OpenMP support In-Reply-To: References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D79CD3C.6050004@behnel.de> <4D79D5C3.7080805@student.matnat.uio.no> Message-ID: <4D7BFA76.80100@canterbury.ac.nz> mark florisson wrote: > Have we ever thought about supporting 'with gil' as an actual > statement instead of just as part of a function declaration or > definition? That's the way I was originally going to do it in Pyrex, but it turned out to be problematic, because there is some setup that gets done on entering a function -- increfing parameters, initialising locals to None, etc. -- that is needed in order to support Python operations being done anywhere in the function. Allowing for a 'with gil' block would require deferring those things until entering the block, and making sure nothing is done outside the block that relies on them. Probably not impossible, but it would require some extensive rearrangements. -- Greg From stefan_ml at behnel.de Sun Mar 13 16:44:59 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 13 Mar 2011 16:44:59 +0100 Subject: [Cython] OpenMP support In-Reply-To: <4D7BFA76.80100@canterbury.ac.nz> References: <4D766C57.2090806@behnel.de> <201103082016.02413.faltet@pytables.org> <4D79CD3C.6050004@behnel.de> <4D79D5C3.7080805@student.matnat.uio.no> <4D7BFA76.80100@canterbury.ac.nz> Message-ID: <4D7CE67B.70306@behnel.de> Greg Ewing, 12.03.2011 23:57: > mark florisson wrote: > >> Have we ever thought about supporting 'with gil' as an actual >> statement instead of just as part of a function declaration or >> definition? > > That's the way I was originally going to do it in Pyrex, but > it turned out to be problematic, because there is some setup > that gets done on entering a function -- increfing parameters, > initialising locals to None, etc. -- that is needed in order > to support Python operations being done anywhere in the function. > > Allowing for a 'with gil' block would require deferring those > things until entering the block, and making sure nothing is done > outside the block that relies on them. Probably not impossible, > but it would require some extensive rearrangements. Yes, absolutely. We may have come a lot closer now that Vitja is working on control flow analysis. That will allow us to see where variables need to get initialised and to decide what needs to be done when entering and leaving a "with gil" block. It may still not be trivial even with that, though. Stefan From markflorisson88 at gmail.com Wed Mar 16 11:28:08 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 11:28:08 +0100 Subject: [Cython] 'with gil:' statement Message-ID: I implemented the 'with gil:' statement, and have added error checking for nested 'with gil' or 'with nogil' statements. For instance, with the patch applied Cython wil issue an error when you have e.g. with nogil: with nogil: ... (or the same thing for 'with gil:'). This because nested 'nogil' or 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort the Python interpreter. However, if people are acquiring the GIL manually in nogil blocks and then want to release the GIL with a 'nogil' block, it will incorrectly issue an error. I do think this kind of code is uncommon, but perhaps we want to issue a warning instead? The 'with gil:' statement can now be used in the same way as 'with nogil:'. Exceptions raised from GIL blocks will be propagated if possible (i.e., if the return type is 'object'). Otherwise it will jump to the end of the function and use the usual __Pyx_WriteUnraisable, there's not really anything new there. For functions declared 'nogil' that contain 'with gil:' statements, it will safely lock around around the initialization of any Python objects and set up the refnanny context (with appropriate preprocessor guards). At the end of the function it will safely lock around the teardown of the refnanny context. With 'safely' I mean that it will create a thread state if it was not already created and may be called even while the GIL is already held (using PyGILState_Ensure()). This means variables are declared and initialized in the same way as in normal GIL-holding functions (except that there is additional locking), and of course the GIL-checking code ensures that errors are issued if those variables are attempted to be used outside any GIL blocks. Could someone review the patch (which is attached)? Maybe check if I haven't missed any side cases and such? -------------- next part -------------- A non-text attachment was scrubbed... Name: with_gil_statement.patch Type: application/octet-stream Size: 14887 bytes Desc: not available URL: From d.s.seljebotn at astro.uio.no Wed Mar 16 11:58:28 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 11:58:28 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: Message-ID: <4D8097D4.5070801@student.matnat.uio.no> On 03/16/2011 11:28 AM, mark florisson wrote: > I implemented the 'with gil:' statement, and have added error checking > for nested 'with gil' or 'with nogil' statements. For instance, with > the patch applied Cython wil issue an error when you have e.g. > > with nogil: > with nogil: > ... > > (or the same thing for 'with gil:'). This because nested 'nogil' or > 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort > the Python interpreter. However, if people are acquiring the GIL > manually in nogil blocks and then want to release the GIL with a > 'nogil' block, it will incorrectly issue an error. I do think this > kind of code is uncommon, but perhaps we want to issue a warning > instead? I think we should make nested nogil-s noops, i.e. with nogil: with nogil: # => if True: This is because one may want to change "with nogil" to "with gil" for debugging purposes (allow printing debug information). Another feedback is that I wonder whether we should put the "gil" and "nogil" psuedo-context managers both in cython namespace, and sort of deprecate the "global" nogil, rather than introduce yet another name that can't be used safely for all kinds of variables. -- Dag > The 'with gil:' statement can now be used in the same way as 'with > nogil:'. Exceptions raised from GIL blocks will be propagated if > possible (i.e., if the return type is 'object'). Otherwise it will > jump to the end of the function and use the usual > __Pyx_WriteUnraisable, there's not really anything new there. > > For functions declared 'nogil' that contain 'with gil:' statements, it > will safely lock around around the initialization of any Python > objects and set up the refnanny context (with appropriate preprocessor > guards). At the end of the function it will safely lock around the > teardown of the refnanny context. With 'safely' I mean that it will > create a thread state if it was not already created and may be called > even while the GIL is already held (using PyGILState_Ensure()). This > means variables are declared and initialized in the same way as in > normal GIL-holding functions (except that there is additional > locking), and of course the GIL-checking code ensures that errors are > issued if those variables are attempted to be used outside any GIL > blocks. > > Could someone review the patch (which is attached)? Maybe check if I > haven't missed any side cases and such? > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Wed Mar 16 11:33:30 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 11:33:30 +0100 Subject: [Cython] PyCharm Message-ID: The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the Cython project a free Open Source license, which means that anyone developing Cython may freely use PyCharm to develop Cython. They prefer to license to remain unpublic, so if you develop Cython and want a free PyCharm license, send me an email and I will send you the license key. It is valid until 13 March 2012. Cheers, Mark From markflorisson88 at gmail.com Wed Mar 16 12:54:00 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 12:54:00 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D8097D4.5070801@student.matnat.uio.no> References: <4D8097D4.5070801@student.matnat.uio.no> Message-ID: On 16 March 2011 11:58, Dag Sverre Seljebotn wrote: > On 03/16/2011 11:28 AM, mark florisson wrote: > > I implemented the 'with gil:' statement, and have added error checking > for nested 'with gil' or 'with nogil' statements. For instance, with > the patch applied Cython wil issue an error when you have e.g. > > with nogil: > with nogil: > ... > > (or the same thing for 'with gil:'). This because nested 'nogil' or > 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort > the Python interpreter. However, if people are acquiring the GIL > manually in nogil blocks and then want to release the GIL with a > 'nogil' block, it will incorrectly issue an error. I do think this > kind of code is uncommon, but perhaps we want to issue a warning > instead? > > I think we should make nested nogil-s noops, i.e. > > with nogil: > ??? with nogil: # => if True: > > This is because one may want to change "with nogil" to "with gil" for > debugging purposes (allow printing debug information). Interesting, that does sound convenient, but I'm not if mere convenience should move us to simply ignore what is technically most likely incorrect code (unless there is intermediate manual locking). In any case, I wouldn't really be against that. If you simply want to allow this for debugging, we could also allow print statements in nogil sections, by either rewriting it using 'with gil:', or by inserting a simple printf (in which case you probably want to place a few restrictions). > Another feedback is that I wonder whether we should put the "gil" and > "nogil" psuedo-context managers both in cython namespace, and sort of > deprecate the "global" nogil, rather than introduce yet another name that > can't be used safely for all kinds of variables. Hmm, good catch. Actually, 'with cython.nogil:' is already possible, but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to simply be ignored). So I guess I'll have to fix that :) > -- Dag > > The 'with gil:' statement can now be used in the same way as 'with > nogil:'. Exceptions raised from GIL blocks will be propagated if > possible (i.e., if the return type is 'object'). Otherwise it will > jump to the end of the function and use the usual > __Pyx_WriteUnraisable, there's not really anything new there. > > For functions declared 'nogil' that contain 'with gil:' statements, it > will safely lock around around the initialization of any Python > objects and set up the refnanny context (with appropriate preprocessor > guards). At the end of the function it will safely lock around the > teardown of the refnanny context. With 'safely' I mean that it will > create a thread state if it was not already created and may be called > even while the GIL is already held (using PyGILState_Ensure()). This > means variables are declared and initialized in the same way as in > normal GIL-holding functions (except that there is additional > locking), and of course the GIL-checking code ensures that errors are > issued if those variables are attempted to be used outside any GIL > blocks. > > Could someone review the patch (which is attached)? Maybe check if I > haven't missed any side cases and such? > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > From markflorisson88 at gmail.com Wed Mar 16 13:01:04 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 13:01:04 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> Message-ID: >> Another feedback is that I wonder whether we should put the "gil" and >> "nogil" psuedo-context managers both in cython namespace, and sort of >> deprecate the "global" nogil, rather than introduce yet another name that >> can't be used safely for all kinds of variables. > > Hmm, good catch. Actually, 'with cython.nogil:' is already possible, > but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to > simply be ignored). > So I guess I'll have to fix that :) > I attached a patch for 'with cython.gil:'. -------------- next part -------------- A non-text attachment was scrubbed... Name: with_cython.gil_statement.patch Type: application/octet-stream Size: 1379 bytes Desc: not available URL: From markflorisson88 at gmail.com Wed Mar 16 13:28:26 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 13:28:26 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> Message-ID: On 16 March 2011 13:01, mark florisson wrote: >>> Another feedback is that I wonder whether we should put the "gil" and >>> "nogil" psuedo-context managers both in cython namespace, and sort of >>> deprecate the "global" nogil, rather than introduce yet another name that >>> can't be used safely for all kinds of variables. >> >> Hmm, good catch. Actually, 'with cython.nogil:' is already possible, >> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to >> simply be ignored). >> So I guess I'll have to fix that :) >> > > I attached a patch for 'with cython.gil:'. > Attached a patch to disallow invalid directives in with statements. I ran the test suite to ensure I didn't break anything. -------------- next part -------------- A non-text attachment was scrubbed... Name: error_on_invalid_with_directives.patch Type: application/octet-stream Size: 985 bytes Desc: not available URL: From d.s.seljebotn at astro.uio.no Wed Mar 16 13:37:19 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 13:37:19 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> Message-ID: <4D80AEFF.4050403@student.matnat.uio.no> On 03/16/2011 12:54 PM, mark florisson wrote: > On 16 March 2011 11:58, Dag Sverre Seljebotn wrote: >> On 03/16/2011 11:28 AM, mark florisson wrote: >> >> I implemented the 'with gil:' statement, and have added error checking >> for nested 'with gil' or 'with nogil' statements. For instance, with >> the patch applied Cython wil issue an error when you have e.g. >> >> with nogil: >> with nogil: >> ... >> >> (or the same thing for 'with gil:'). This because nested 'nogil' or >> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort >> the Python interpreter. However, if people are acquiring the GIL >> manually in nogil blocks and then want to release the GIL with a >> 'nogil' block, it will incorrectly issue an error. I do think this >> kind of code is uncommon, but perhaps we want to issue a warning >> instead? >> >> I think we should make nested nogil-s noops, i.e. >> >> with nogil: >> with nogil: # => if True: >> >> This is because one may want to change "with nogil" to "with gil" for >> debugging purposes (allow printing debug information). > Interesting, that does sound convenient, but I'm not if mere > convenience should move us to simply ignore what is technically most > likely incorrect code (unless there is intermediate manual locking). I'm not sure if I understand what you mean here. How does simply ignoring redundant "with [no]gil" statements cause incorrect code? Or do you mean this is a I'm just trying to minimize the "language getting in your way" factor. It is pretty useless to write if x: if x: ... as well, but Python does allow it. Warnings on nested "with nogil" is more the role of a "cylint" in my opinion. > In any case, I wouldn't really be against that. If you simply want to > allow this for debugging, we could also allow print statements in > nogil sections, by either rewriting it using 'with gil:', or by > inserting a simple printf (in which case you probably want to place a > few restrictions). It's not only print statements. I.e., if I think something is wrong with an array, I'll stick in code like print np.std(x), np.mean(x), np.any(np.isnan(x)) or something more complex that may require temporaries. Or even plot the vector: plt.plot(x) plt.show() # blocks until I close plot window Or, launch a debugger: if np.any(np.isnan(x)): import pdb; pdb.set_trace() so I guess I should stop saying this is only about printing. In general, it's nice to use Python during debugging. I find myself replacing "with nogil" with "if True" in such situations, to avoid reindenting. I guess I can soon start using "with gil" around the debug code though. Again, the restriction on nested nogil/gil is not a big problem, just an instance of "the language getting in your way". Dag From markflorisson88 at gmail.com Wed Mar 16 13:55:44 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 13:55:44 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D80AEFF.4050403@student.matnat.uio.no> References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> Message-ID: On 16 March 2011 13:37, Dag Sverre Seljebotn wrote: > On 03/16/2011 12:54 PM, mark florisson wrote: >> >> On 16 March 2011 11:58, Dag Sverre Seljebotn >> ?wrote: >>> >>> On 03/16/2011 11:28 AM, mark florisson wrote: >>> >>> I implemented the 'with gil:' statement, and have added error checking >>> for nested 'with gil' or 'with nogil' statements. For instance, with >>> the patch applied Cython wil issue an error when you have e.g. >>> >>> with nogil: >>> ? ? with nogil: >>> ? ? ? ? ... >>> >>> (or the same thing for 'with gil:'). This because nested 'nogil' or >>> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort >>> the Python interpreter. However, if people are acquiring the GIL >>> manually in nogil blocks and then want to release the GIL with a >>> 'nogil' block, it will incorrectly issue an error. I do think this >>> kind of code is uncommon, but perhaps we want to issue a warning >>> instead? >>> >>> I think we should make nested nogil-s noops, i.e. >>> >>> with nogil: >>> ? ? with nogil: # => ?if True: >>> >>> This is because one may want to change "with nogil" to "with gil" for >>> debugging purposes (allow printing debug information). >> >> Interesting, that does sound convenient, but I'm not if mere >> convenience should move us to simply ignore what is technically most >> likely incorrect code (unless there is intermediate manual locking). > > I'm not sure if I understand what you mean here. How does simply ignoring > redundant "with [no]gil" statements cause incorrect code? Or do you mean > this is a > > I'm just trying to minimize the "language getting in your way" factor. It is > pretty useless to write > > if x: > ? ?if x: > ? ? ? ?... > > as well, but Python does allow it. > > Warnings on nested "with nogil" is more the role of a "cylint" in my > opinion. > Perhaps you're right. However, I just think it is important for users to realize that in general, they cannot unblock threads recursively. Currently the error checking code catches multiple nested 'with (no)gil', but it doesn't catch this: cdef void func() nogil: with nogil: pass with nogil: func() But the problem is that it does abort the interpreter. So I thought that perhaps emphasizing that that code is incorrect for at least the easy-to-catch cases, we might make users somewhat more aware. Because if the above code aborts Python, but a nested 'with nogil:' is valid code, there might be a source for confusion. >> In any case, I wouldn't really be against that. If you simply want to >> allow this for debugging, we could also allow print statements in >> nogil sections, by either rewriting it using 'with gil:', or by >> inserting a simple printf (in which case you probably want to place a >> few restrictions). > > It's not only print statements. I.e., if I think something is wrong with an > array, I'll stick in code like > > print np.std(x), np.mean(x), np.any(np.isnan(x)) > > or something more complex that may require temporaries. Or even plot the > vector: > > plt.plot(x) > plt.show() # blocks until I close plot window > > Or, launch a debugger: > > if np.any(np.isnan(x)): > ? ? import pdb; pdb.set_trace() > > so I guess I should stop saying this is only about printing. In general, > it's nice to use Python during debugging. I find myself replacing "with > nogil" with "if True" in such situations, to avoid reindenting. > > I guess I can soon start using "with gil" around the debug code though. > Again, the restriction on nested nogil/gil is not a big problem, just an > instance of "the language getting in your way". I understand. If your code doesn't introduce Cython-level blocks, you could at least put it on one line so you can easily comment it out. > Dag > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Wed Mar 16 13:59:04 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 13:59:04 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> Message-ID: On 16 March 2011 13:55, mark florisson wrote: > On 16 March 2011 13:37, Dag Sverre Seljebotn wrote: >> On 03/16/2011 12:54 PM, mark florisson wrote: >>> >>> On 16 March 2011 11:58, Dag Sverre Seljebotn >>> ?wrote: >>>> >>>> On 03/16/2011 11:28 AM, mark florisson wrote: >>>> >>>> I implemented the 'with gil:' statement, and have added error checking >>>> for nested 'with gil' or 'with nogil' statements. For instance, with >>>> the patch applied Cython wil issue an error when you have e.g. >>>> >>>> with nogil: >>>> ? ? with nogil: >>>> ? ? ? ? ... >>>> >>>> (or the same thing for 'with gil:'). This because nested 'nogil' or >>>> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort >>>> the Python interpreter. However, if people are acquiring the GIL >>>> manually in nogil blocks and then want to release the GIL with a >>>> 'nogil' block, it will incorrectly issue an error. I do think this >>>> kind of code is uncommon, but perhaps we want to issue a warning >>>> instead? >>>> >>>> I think we should make nested nogil-s noops, i.e. >>>> >>>> with nogil: >>>> ? ? with nogil: # => ?if True: >>>> >>>> This is because one may want to change "with nogil" to "with gil" for >>>> debugging purposes (allow printing debug information). >>> >>> Interesting, that does sound convenient, but I'm not if mere >>> convenience should move us to simply ignore what is technically most >>> likely incorrect code (unless there is intermediate manual locking). >> >> I'm not sure if I understand what you mean here. How does simply ignoring >> redundant "with [no]gil" statements cause incorrect code? Or do you mean >> this is a >> >> I'm just trying to minimize the "language getting in your way" factor. It is >> pretty useless to write >> >> if x: >> ? ?if x: >> ? ? ? ?... >> >> as well, but Python does allow it. >> >> Warnings on nested "with nogil" is more the role of a "cylint" in my >> opinion. >> > > Perhaps you're right. However, I just think it is important for users > to realize that in general, they cannot unblock threads recursively. > Currently the error checking code catches multiple nested 'with > (no)gil', but it doesn't catch this: > > cdef void func() nogil: > ? ?with nogil: > ? ? ? ?pass > > with nogil: > ? ?func() > > But the problem is that it does abort the interpreter. So I thought > that perhaps emphasizing that that code is incorrect for at least the > easy-to-catch cases, we might make users somewhat more aware. Because > if the above code aborts Python, but a nested 'with nogil:' is valid > code, there might be a source for confusion. > I have to mention, though, that the converse is not true. The gil may be acquired recursively, i.e., 'with gil:' blocks may be nested. From d.s.seljebotn at astro.uio.no Wed Mar 16 14:10:29 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 14:10:29 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> Message-ID: <4D80B6C5.2020509@student.matnat.uio.no> On 03/16/2011 01:55 PM, mark florisson wrote: > On 16 March 2011 13:37, Dag Sverre Seljebotn wrote: >> On 03/16/2011 12:54 PM, mark florisson wrote: >>> On 16 March 2011 11:58, Dag Sverre Seljebotn >>> wrote: >>>> On 03/16/2011 11:28 AM, mark florisson wrote: >>>> >>>> I implemented the 'with gil:' statement, and have added error checking >>>> for nested 'with gil' or 'with nogil' statements. For instance, with >>>> the patch applied Cython wil issue an error when you have e.g. >>>> >>>> with nogil: >>>> with nogil: >>>> ... >>>> >>>> (or the same thing for 'with gil:'). This because nested 'nogil' or >>>> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort >>>> the Python interpreter. However, if people are acquiring the GIL >>>> manually in nogil blocks and then want to release the GIL with a >>>> 'nogil' block, it will incorrectly issue an error. I do think this >>>> kind of code is uncommon, but perhaps we want to issue a warning >>>> instead? >>>> >>>> I think we should make nested nogil-s noops, i.e. >>>> >>>> with nogil: >>>> with nogil: # => if True: >>>> >>>> This is because one may want to change "with nogil" to "with gil" for >>>> debugging purposes (allow printing debug information). >>> Interesting, that does sound convenient, but I'm not if mere >>> convenience should move us to simply ignore what is technically most >>> likely incorrect code (unless there is intermediate manual locking). >> I'm not sure if I understand what you mean here. How does simply ignoring >> redundant "with [no]gil" statements cause incorrect code? Or do you mean >> this is a >> >> I'm just trying to minimize the "language getting in your way" factor. It is >> pretty useless to write >> >> if x: >> if x: >> ... >> >> as well, but Python does allow it. >> >> Warnings on nested "with nogil" is more the role of a "cylint" in my >> opinion. >> > Perhaps you're right. However, I just think it is important for users > to realize that in general, they cannot unblock threads recursively. > Currently the error checking code catches multiple nested 'with > (no)gil', but it doesn't catch this: > > cdef void func() nogil: > with nogil: > pass > > with nogil: > func() > > But the problem is that it does abort the interpreter. So I thought > that perhaps emphasizing that that code is incorrect for at least the > easy-to-catch cases, we might make users somewhat more aware. Because > if the above code aborts Python, but a nested 'with nogil:' is valid > code, there might be a source for confusion. Ah, right. I guess I agree with disallowing nested "with nogil" statements for the time being then. Dag Sverre From stefan_ml at behnel.de Wed Mar 16 15:07:15 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Mar 2011 15:07:15 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> Message-ID: <4D80C413.8080207@behnel.de> mark florisson, 16.03.2011 13:28: > On 16 March 2011 13:01, mark florisson wrote: >>>> Another feedback is that I wonder whether we should put the "gil" and >>>> "nogil" psuedo-context managers both in cython namespace, and sort of >>>> deprecate the "global" nogil, rather than introduce yet another name that >>>> can't be used safely for all kinds of variables. >>> >>> Hmm, good catch. Actually, 'with cython.nogil:' is already possible, >>> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to >>> simply be ignored). >>> So I guess I'll have to fix that :) >>> >> >> I attached a patch for 'with cython.gil:'. Looks ok. > Attached a patch to disallow invalid directives in with statements. +1, except that I'd print the directive name as part of the error message. Stefan From stefan_ml at behnel.de Wed Mar 16 15:11:16 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Mar 2011 15:11:16 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D8097D4.5070801@student.matnat.uio.no> References: <4D8097D4.5070801@student.matnat.uio.no> Message-ID: <4D80C504.7050300@behnel.de> Dag Sverre Seljebotn, 16.03.2011 11:58: > On 03/16/2011 11:28 AM, mark florisson wrote: >> I implemented the 'with gil:' statement, and have added error checking >> for nested 'with gil' or 'with nogil' statements. For instance, with >> the patch applied Cython wil issue an error when you have e.g. >> >> with nogil: >> with nogil: >> ... >> >> (or the same thing for 'with gil:'). This because nested 'nogil' or >> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort >> the Python interpreter. However, if people are acquiring the GIL >> manually in nogil blocks and then want to release the GIL with a >> 'nogil' block, it will incorrectly issue an error. I do think this >> kind of code is uncommon, but perhaps we want to issue a warning >> instead? > > I think we should make nested nogil-s noops, i.e. > > with nogil: > with nogil: # => if True: -1, the compiler should bark at clearly nonsensical code. The above, if encountered in the wild, is most likely a copy&paste bug that wasn't intended that way. (ok, you'd most likely get other errors as well due to GIL checking, but that doesn't change the fact that the code is nonsensical) Stefan From stefan_ml at behnel.de Wed Mar 16 15:15:41 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Mar 2011 15:15:41 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D80AEFF.4050403@student.matnat.uio.no> References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> Message-ID: <4D80C60D.1080605@behnel.de> Dag Sverre Seljebotn, 16.03.2011 13:37: > On 03/16/2011 12:54 PM, mark florisson wrote: >> On 16 March 2011 11:58, Dag Sverre Seljebotn wrote: >>> I think we should make nested nogil-s noops, i.e. >>> >>> with nogil: >>> with nogil: # => if True: >>> >>> This is because one may want to change "with nogil" to "with gil" for >>> debugging purposes (allow printing debug information). >> Interesting, that does sound convenient, but I'm not if mere >> convenience should move us to simply ignore what is technically most >> likely incorrect code (unless there is intermediate manual locking). > > I'm just trying to minimize the "language getting in your way" factor. It > is pretty useless to write > > if x: > if x: > ... > > as well, but Python does allow it. That's because it's not necessarily useless. It can have side-effects. >> In any case, I wouldn't really be against that. If you simply want to >> allow this for debugging, we could also allow print statements in >> nogil sections, by either rewriting it using 'with gil:', or by >> inserting a simple printf (in which case you probably want to place a >> few restrictions). > > It's not only print statements. I.e., if I think something is wrong with an > array, I'll stick in code like > > print np.std(x), np.mean(x), np.any(np.isnan(x)) > > or something more complex that may require temporaries. Or even plot the > vector: > > plt.plot(x) > plt.show() # blocks until I close plot window > > Or, launch a debugger: > > if np.any(np.isnan(x)): > import pdb; pdb.set_trace() All of these are better expressed using an explicit "with gil", also in debug code. Stefan From stefan_ml at behnel.de Wed Mar 16 15:26:56 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Mar 2011 15:26:56 +0100 Subject: [Cython] PyCharm In-Reply-To: References: Message-ID: <4D80C8B0.4010900@behnel.de> mark florisson, 16.03.2011 11:33: > The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the > Cython project a free Open Source license, which means that anyone > developing Cython may freely use PyCharm to develop Cython. Looks like they don't support Cython code, though: http://youtrack.jetbrains.net/issue/PY-1046 (although, maybe that's the reason for the offer ;) Stefan From markflorisson88 at gmail.com Wed Mar 16 15:32:45 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 15:32:45 +0100 Subject: [Cython] PyCharm In-Reply-To: <4D80C8B0.4010900@behnel.de> References: <4D80C8B0.4010900@behnel.de> Message-ID: On 16 March 2011 15:26, Stefan Behnel wrote: > mark florisson, 16.03.2011 11:33: >> >> The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the >> Cython project a free Open Source license, which means that anyone >> developing Cython may freely use PyCharm to develop Cython. > > Looks like they don't support Cython code, though: > > http://youtrack.jetbrains.net/issue/PY-1046 > > (although, maybe that's the reason for the offer ;) > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > Indeed, that is rather unfortunate. I only use it for plain Python code, so I work with two editors, which is somewhat inconvenient. From markflorisson88 at gmail.com Wed Mar 16 15:29:16 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 15:29:16 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D80C413.8080207@behnel.de> References: <4D8097D4.5070801@student.matnat.uio.no> <4D80C413.8080207@behnel.de> Message-ID: On 16 March 2011 15:07, Stefan Behnel wrote: > mark florisson, 16.03.2011 13:28: >> >> On 16 March 2011 13:01, mark florisson wrote: >>>>> >>>>> Another feedback is that I wonder whether we should put the "gil" and >>>>> "nogil" psuedo-context managers both in cython namespace, and sort of >>>>> deprecate the "global" nogil, rather than introduce yet another name >>>>> that >>>>> can't be used safely for all kinds of variables. >>>> >>>> Hmm, good catch. Actually, 'with cython.nogil:' is already possible, >>>> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to >>>> simply be ignored). >>>> So I guess I'll have to fix that :) >>>> >>> >>> I attached a patch for 'with cython.gil:'. > > Looks ok. Does that mean I should push the changes, or do you want some more elaborate reviewing first? > >> Attached a patch to disallow invalid directives in with statements. > > +1, except that I'd print the directive name as part of the error message. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Wed Mar 16 15:53:35 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Mar 2011 15:53:35 +0100 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> Message-ID: <4D80CEEF.4080701@behnel.de> Robert Bradshaw, 11.03.2011 19:33: > On Fri, Mar 11, 2011 at 9:36 AM, Stefan Behnel wrote: >> Stefan Behnel, 11.03.2011 15:08: >>> >>> Vitja Makarov, 11.03.2011 15:04: >>>> >>>> 2011/3/11 Stefan Behnel: >>>>> >>>>> Personally, I think it would be nice to keep up Python's semantics, but >>>>> when >>>>> I implemented this I broke quite some code in Sage (you may have noticed >>>>> that the sage-build project in Hudson has been red for a while). There >>>>> are >>>>> things in C and especially in C++ that cannot be easily copied into a >>>>> temporary variable in order to make sure they are evaluated before the >>>>> following arguments. This is not a problem for Python function calls >>>>> where >>>>> all arguments end up being copied (and often converted) anyway. It is a >>>>> problem for C function calls, though. >>>> >>>>> f(g(a), a.x, h(a)) >>>> >>>> Why could not this be translated into: >>>> >>>> tmp1 = g(a) >>>> tmp2 = a.x >>>> tmp3 = h(a) >>>> >>>> f(tmp1, tmp2, tmp3) >>> >>> See above. >> >> To be a little clearer here, it's a problem in C for example with struct >> values. Copying them by value into a temp variable can be expensive, >> potentially twice as expensive as simply passing them into the function >> normally. > > Yep, and some types (e.g. array types) can't be assigned to at all. > FWIW, the issues with Sage is that many libraries use the "stack > allocated, pass-by-reference" trick > > typedef foo_struct foo_t[1] > > but we have declared them to be of type "void*" because we don't care > about or want to muck with the internals. Cleaning this up is > something we should do, but is taking a while, and aside from that it > makes Cython even more dependent on correct type declarations (and > backwards incompatible in this regard). Sage is a great test for the > buildbot, so keeping it red for so long is not a good thing either. > >> Not sure what kind of additional devilry C++ provides here, but I'd expect >> that object values can exhibit bizarre behaviour when being assigned. Maybe >> others can enlighten me here. > > Yes, C++ allows overloading of the assignment operator, so assigning > may lead to arbitrary code (as well as probably an expensive copy, as > with structs, and structs in C++ are really just classes with a > different visibility). > >> I have no idea how many cases there actually are that we can't handle or >> that may lead to a performance degradation when using temps, but the problem >> is that they exist at all. > > Note that this applies not just to function arguments, but a host of > other places, e.g. the order of evaluation of "f() + g()" in C is > unspecified. Fleshing this out completely will lead to a lot more > temps and verbose C code. And then we'd just cross our fingers that > the C compiler was able to optimize all these temps away (though still > possibly producing inferior code if it were able to switch order of > execution). > > Whatever we do, we need a flag/directive. Perhaps there's a way to > guarantee correct ordering for all valid Python code Plain Python code should never suffer from the original problem (with the obvious exception of cpdef functions), but I also see how type inference can affect this guarantee for the evaluation order in expressions. I agree with Greg, though, that it has a code smell if (multiple) subexpressions in an expression have side-effects, especially when they are impacted by the evaluation order. That means that the average innocent future maintainer could break the code by manually rearranging the expression in a way that's perfectly valid mathematically, e.g. as a readability fix. > even in the > presence of (function and variable) type inference, but allow some > leeway for explicitly declared C functions. I'm actually leaning towards not guaranteeing the order of execution if C doesn't do it either. If this is really required, it's easy to work around for users, but it's severely hard to fix for Cython in all cases, and the gain is truly small. After all, we'd only make it easier for users to write bad code. Stefan From pav at iki.fi Wed Mar 16 14:17:59 2011 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 16 Mar 2011 13:17:59 +0000 (UTC) Subject: [Cython] 'with gil:' statement References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> <4D80B6C5.2020509@student.matnat.uio.no> Message-ID: Wed, 16 Mar 2011 14:10:29 +0100, Dag Sverre Seljebotn wrote: > Ah, right. I guess I agree with disallowing nested "with nogil" > statements for the time being then. Could you make the inner nested "with nogil" statements no-ops instead, if the GIL is already released? Does the Cython compiler keep track if GIL is acquired or not? -- Pauli Virtanen From stefan_ml at behnel.de Wed Mar 16 16:14:25 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Mar 2011 16:14:25 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: Message-ID: <4D80D3D1.6000601@behnel.de> mark florisson, 16.03.2011 11:28: > I implemented the 'with gil:' statement > [...] > The 'with gil:' statement can now be used in the same way as 'with > nogil:'. Exceptions raised from GIL blocks will be propagated if > possible (i.e., if the return type is 'object'). Otherwise it will > jump to the end of the function and use the usual > __Pyx_WriteUnraisable, there's not really anything new there. I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. Would you do the same when calling a cdef function that uses "with gil" in its signature? We don't currently do that, but it feels like it's the same situation. > For functions declared 'nogil' that contain 'with gil:' statements, it > will safely lock around around the initialization of any Python > objects and set up the refnanny context (with appropriate preprocessor > guards). At the end of the function it will safely lock around the > teardown of the refnanny context. With 'safely' I mean that it will > create a thread state if it was not already created and may be called > even while the GIL is already held (using PyGILState_Ensure()). This > means variables are declared and initialized in the same way as in > normal GIL-holding functions (except that there is additional > locking), and of course the GIL-checking code ensures that errors are > issued if those variables are attempted to be used outside any GIL > blocks. I find that surprising semantics. So the GIL will always be acquired at least twice in the following example, regardless of the input? cdef int callme(bint flag) nogil: if flag: with gil: x = object() > Could someone review the patch (which is attached)? Maybe check if I > haven't missed any side cases and such? From a first look, the test file you added seems far too short. I would expect that this feature requires a lot more testing in combination with declared and undeclared local variables, type inference, several exception raising and catching situations (e.g. raise in one block, catch in an outer block, try-finally, ...) or looping. It may also have an impact on Vitja's control flow analysis branch that's worth considering. Stefan From stefan_ml at behnel.de Wed Mar 16 16:14:39 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Mar 2011 16:14:39 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> <4D80C413.8080207@behnel.de> Message-ID: <4D80D3DF.3050708@behnel.de> mark florisson, 16.03.2011 15:29: > On 16 March 2011 15:07, Stefan Behnel wrote: >> mark florisson, 16.03.2011 13:28: >>> >>> On 16 March 2011 13:01, mark florisson wrote: >>>>>> >>>>>> Another feedback is that I wonder whether we should put the "gil" and >>>>>> "nogil" psuedo-context managers both in cython namespace, and sort of >>>>>> deprecate the "global" nogil, rather than introduce yet another name >>>>>> that >>>>>> can't be used safely for all kinds of variables. >>>>> >>>>> Hmm, good catch. Actually, 'with cython.nogil:' is already possible, >>>>> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to >>>>> simply be ignored). >>>>> So I guess I'll have to fix that :) >>>>> >>>> >>>> I attached a patch for 'with cython.gil:'. >> >> Looks ok. > > Does that mean I should push the changes, or do you want some more > elaborate reviewing first? I was only commenting on the two simpler follow-up changes. Stefan From d.s.seljebotn at astro.uio.no Wed Mar 16 16:27:40 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 16 Mar 2011 16:27:40 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> <4D80B6C5.2020509@student.matnat.uio.no> Message-ID: <4D80D6EC.90108@student.matnat.uio.no> On 03/16/2011 02:17 PM, Pauli Virtanen wrote: > Wed, 16 Mar 2011 14:10:29 +0100, Dag Sverre Seljebotn wrote: >> Ah, right. I guess I agree with disallowing nested "with nogil" >> statements for the time being then. > Could you make the inner nested "with nogil" statements no-ops instead, > if the GIL is already released? Does the Cython compiler keep track if > GIL is acquired or not? That's what I initially suggested. See Mark's posted code -- when calling another function, you can't know whether the nogil is "nested" or not (without actually checking with CPython...) Within-Cython solutions are bad because calls may cross between different Cython modules. Dag From markflorisson88 at gmail.com Wed Mar 16 17:17:08 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 17:17:08 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D80D3D1.6000601@behnel.de> References: <4D80D3D1.6000601@behnel.de> Message-ID: On 16 March 2011 16:14, Stefan Behnel wrote: > mark florisson, 16.03.2011 11:28: >> >> I implemented the 'with gil:' statement >> [...] >> The 'with gil:' statement can now be used in the same way as 'with >> nogil:'. Exceptions raised from GIL blocks will be propagated if >> possible (i.e., if the return type is 'object'). Otherwise it will >> jump to the end of the function and use the usual >> __Pyx_WriteUnraisable, there's not really anything new there. > > I'm not sure if this is a good idea. "nogil" blocks don't have a way to > handle exceptions, so simply jumping out of them because an inner 'with gil' > block raised an exception can have unexpected side effects. > > Would you do the same when calling a cdef function that uses "with gil" in > its signature? We don't currently do that, but it feels like it's the same > situation. > I beg to differ. Consider this code: with nogil: with gil: raise SomeError printf("something\n") With normal 'with' statements the printf (in the snippet above) would be skipped, and so it is with 'with gil:' blocks, it follows the exact semantics of the Python behaviour as you expect it. With statements are like try/finally, and the 'with (no)gil:' blocks are implemented as such. The above would translate to: try: release gil try: acquire gil raise SomeError finally: release gil printf("something\n") finally: acquire gil If users need to do (C-level) cleanup, they need to wrap the code in the 'with gil:' in something like try/finally, or perhaps try/except. In fact, if you declare a function 'with gil', you will want to either handle any exception in that function, or you want your caller to check for errors while returning an error indicator. So in that case, you should still do error checking, if you don't, exceptions will simply be ignored. Another related issue, you cannot declare your function with both 'with gil' and 'except NULL' at the same time, as calling such a function without having the GIL will segfault your program. In fact, I think we should add a check for this, as e.g. the following code will segfault in func() when it calls __Pyx_WriteUnraisable, as it will try to get the exception from a NULL thread-state. from cpython.ref cimport PyObject from libc.stdio cimport printf cdef PyObject *func2() except NULL with gil: raise Exception("blah") cdef void func() nogil: cdef PyObject *p = func2() printf("This is not printed!\n") func() Of course, you cannot really declare the return type of func2 as 'object', because you wouldn't be able to call it from 'nogil' sections that way. I guess my point is, if the user isn't very careful about the way exceptions are handled, undesired behaviour may occur. With both functions and with gil blocks, the user will have to override the default behaviour if needed. I think "python semantics" most accurately reflect the syntax for 'with gil:' blocks. >> For functions declared 'nogil' that contain 'with gil:' statements, it >> will safely lock around around the initialization of any Python >> objects and set up the refnanny context (with appropriate preprocessor >> guards). At the end of the function it will safely lock around the >> teardown of the refnanny context. With 'safely' I mean that it will >> create a thread state if it was not already created and may be called >> even while the GIL is already held (using PyGILState_Ensure()). This >> means variables are declared and initialized in the same way as in >> normal GIL-holding functions (except that there is additional >> locking), and of course the GIL-checking code ensures that errors are >> issued if those variables are attempted to be used outside any GIL >> blocks. > > I find that surprising semantics. So the GIL will always be acquired at > least twice in the following example, regardless of the input? > > ? ?cdef int callme(bint flag) nogil: > ? ? ? ? if flag: > ? ? ? ? ? ? with gil: x = object() > Yes. It should be noted to users that 'with gil:' does not come without its repercussions. >> Could someone review the patch (which is attached)? Maybe check if I >> haven't missed any side cases and such? > > From a first look, the test file you added seems far too short. I would > expect that this feature requires a lot more testing in combination with > declared and undeclared local variables, type inference, several exception > raising and catching situations (e.g. raise in one block, catch in an outer > block, try-finally, ...) or looping. It may also have an impact on Vitja's > control flow analysis branch that's worth considering. I agree. I think I'll start a branch and work on some more tests. > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Wed Mar 16 20:16:38 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 16 Mar 2011 20:16:38 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D80D3D1.6000601@behnel.de> Message-ID: >>> Could someone review the patch (which is attached)? Maybe check if I >>> haven't missed any side cases and such? >> >> From a first look, the test file you added seems far too short. I would >> expect that this feature requires a lot more testing in combination with >> declared and undeclared local variables, type inference, several exception >> raising and catching situations (e.g. raise in one block, catch in an outer >> block, try-finally, ...) or looping. It may also have an impact on Vitja's >> control flow analysis branch that's worth considering. > > I agree. I think I'll start a branch and work on some more tests. I also have to adjust the function teardown locking. I'll start a branch and report back if I get positive results. From greg.ewing at canterbury.ac.nz Wed Mar 16 23:28:46 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Mar 2011 11:28:46 +1300 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> Message-ID: <4D81399E.2080904@canterbury.ac.nz> mark florisson wrote: > I just think it is important for users > to realize that in general, they cannot unblock threads recursively. I don't think that nested 'with nogil' blocks in any way suggest that they can. Saying 'with nogil' means "I want this code executed with the gil released". If it's already released, the logical meaning is to do nothing further. -- Greg From greg.ewing at canterbury.ac.nz Wed Mar 16 23:42:11 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Mar 2011 11:42:11 +1300 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D80B6C5.2020509@student.matnat.uio.no> References: <4D8097D4.5070801@student.matnat.uio.no> <4D80AEFF.4050403@student.matnat.uio.no> <4D80B6C5.2020509@student.matnat.uio.no> Message-ID: <4D813CC3.7090102@canterbury.ac.nz> > On 03/16/2011 01:55 PM, mark florisson wrote: > >> but it doesn't catch this: >> >> cdef void func() nogil: >> with nogil: >> pass >> >> with nogil: >> func() I would say that in that case, func() should acquire the gil on entry so that it can be released in the 'with nogil' block. Either that or find some way to generate code that tests whether the gil is currently held and releases it if it is. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 17 00:13:19 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Mar 2011 12:13:19 +1300 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D80CEEF.4080701@behnel.de> References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> Message-ID: <4D81440F.2080205@canterbury.ac.nz> Stefan Behnel wrote: > That means that the average > innocent future maintainer could break the code by manually rearranging > the expression in a way that's perfectly valid mathematically, Another thing is that when the subexpressions concerned are function arguments, it only works if the order of the arguments happens to be the same as the required order of evaluation. Changes in the function signature can therefore lead to subtle breakage. -- Greg From greg.ewing at canterbury.ac.nz Thu Mar 17 00:24:59 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Mar 2011 12:24:59 +1300 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D80D3D1.6000601@behnel.de> References: <4D80D3D1.6000601@behnel.de> Message-ID: <4D8146CB.6060608@canterbury.ac.nz> Stefan Behnel wrote: > I'm not sure if this is a good idea. "nogil" blocks don't have a way to > handle exceptions, so simply jumping out of them because an inner 'with > gil' block raised an exception can have unexpected side effects. Seems to me that the __Pyx_WriteUnraisable should be done at the end of the 'with gil' block, and execution should then continue from there. In other words, the effect on exception handling should be the same as if the 'with gil' block had been factored out into a separate function having no exception return value. -- Greg From robertwb at math.washington.edu Thu Mar 17 05:05:01 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 16 Mar 2011 21:05:01 -0700 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D80CEEF.4080701@behnel.de> References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> Message-ID: On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: > Robert Bradshaw, 11.03.2011 19:33: >> >> On Fri, Mar 11, 2011 at 9:36 AM, Stefan Behnel >> ?wrote: >>> >>> Stefan Behnel, 11.03.2011 15:08: >>>> >>>> Vitja Makarov, 11.03.2011 15:04: >>>>> >>>>> 2011/3/11 Stefan Behnel: >>>>>> >>>>>> Personally, I think it would be nice to keep up Python's semantics, >>>>>> but >>>>>> when >>>>>> I implemented this I broke quite some code in Sage (you may have >>>>>> noticed >>>>>> that the sage-build project in Hudson has been red for a while). There >>>>>> are >>>>>> things in C and especially in C++ that cannot be easily copied into a >>>>>> temporary variable in order to make sure they are evaluated before the >>>>>> following arguments. This is not a problem for Python function calls >>>>>> where >>>>>> all arguments end up being copied (and often converted) anyway. It is >>>>>> a >>>>>> problem for C function calls, though. >>>>> >>>>>> f(g(a), a.x, h(a)) >>>>> >>>>> Why could not this be translated into: >>>>> >>>>> tmp1 = g(a) >>>>> tmp2 = a.x >>>>> tmp3 = h(a) >>>>> >>>>> f(tmp1, tmp2, tmp3) >>>> >>>> See above. >>> >>> To be a little clearer here, it's a problem in C for example with struct >>> values. Copying them by value into a temp variable can be expensive, >>> potentially twice as expensive as simply passing them into the function >>> normally. >> >> Yep, and some types (e.g. array types) can't be assigned to at all. >> FWIW, the issues with Sage is that many libraries use the "stack >> allocated, pass-by-reference" trick >> >> ? ? typedef foo_struct foo_t[1] >> >> but we have declared them to be of type "void*" because we don't care >> about or want to muck with the internals. Cleaning this up is >> something we should do, but is taking a while, and aside from that it >> makes Cython even more dependent on correct type declarations (and >> backwards incompatible in this regard). Sage is a great test for the >> buildbot, so keeping it red for so long is not a good thing either. >> >>> Not sure what kind of additional devilry C++ provides here, but I'd >>> expect >>> that object values can exhibit bizarre behaviour when being assigned. >>> Maybe >>> others can enlighten me here. >> >> Yes, C++ allows overloading of the assignment operator, so assigning >> may lead to arbitrary code (as well as probably an expensive copy, as >> with structs, and structs in C++ are really just classes with a >> different visibility). >> >>> I have no idea how many cases there actually are that we can't handle or >>> that may lead to a performance degradation when using temps, but the >>> problem >>> is that they exist at all. >> >> Note that this applies not just to function arguments, but a host of >> other places, e.g. the order of evaluation of "f() + g()" in C is >> unspecified. Fleshing this out completely will lead to a lot more >> temps and verbose C code. And then we'd just cross our fingers that >> the C compiler was able to optimize all these temps away (though still >> possibly producing inferior code if it were able to switch order of >> execution). >> >> Whatever we do, we need a flag/directive. Perhaps there's a way to >> guarantee correct ordering for all valid Python code > > Plain Python code should never suffer from the original problem (with the > obvious exception of cpdef functions), but I also see how type inference can > affect this guarantee for the evaluation order in expressions. In particular, we need to be careful if we ever automatically cpdef a function (as part of a future optimization). > I agree with Greg, though, that it has a code smell if (multiple) > subexpressions in an expression have side-effects, especially when they are > impacted by the evaluation order. That means that the average innocent > future maintainer could break the code by manually rearranging the > expression in a way that's perfectly valid mathematically, e.g. as a > readability fix. > > >> even in the >> presence of (function and variable) type inference, but allow some >> leeway for explicitly declared C functions. > > I'm actually leaning towards not guaranteeing the order of execution if C > doesn't do it either. If this is really required, it's easy to work around > for users, but it's severely hard to fix for Cython in all cases, and the > gain is truly small. After all, we'd only make it easier for users to write > bad code. Yep. Lets keep the code in for the above case. - Robert From d.s.seljebotn at astro.uio.no Thu Mar 17 08:38:44 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 17 Mar 2011 08:38:44 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D8146CB.6060608@canterbury.ac.nz> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> Message-ID: <4D81BA84.1020409@student.matnat.uio.no> On 03/17/2011 12:24 AM, Greg Ewing wrote: > Stefan Behnel wrote: > >> I'm not sure if this is a good idea. "nogil" blocks don't have a way >> to handle exceptions, so simply jumping out of them because an inner >> 'with gil' block raised an exception can have unexpected side effects. > > Seems to me that the __Pyx_WriteUnraisable should be done at > the end of the 'with gil' block, and execution should then > continue from there. > > In other words, the effect on exception handling should be > the same as if the 'with gil' block had been factored out into > a separate function having no exception return value. > -1. I consider the fact that exceptions don't propagate from some functions a "currently unfixable bug". We should plan for it being fixed some day. Having a "with" statement alter execution flow in this way is totally unintuitive to me. If you want this, it's better to introduce a new keyword like "trywithgil: ... except:" (not that I'm in favour of that). We could perhaps fix exception propagation from nogil functions by using some conventions + setjmp/longjmp. Mono does this when calling into native code, and I recently did it manually in Cython to propagate exceptions through the Fortran wrappers in SciPy. Also, the GIL may not be around forever even in CPython? (All arguments I've seen for keeping it has been along the lines of "it slows down serial code", not that it is considered a good thing.) Designing a language around the GIL feels like a dead-end to me. I'm OK with being practical in the face of the limitations of today; but let's keep "with gil" and "with nogil" something that can become noops in the future without too much pain. Yes, I know that if the GIL goes it will break Stefan's lxml code, and I'm sure other code -- I'm just saying that we shouldn't make the language design even more GIL-centric than it already is. Anyway, I'm off to write some computational Fortran+OpenMP code because dealing with threading and Python is just more than I can deal with... Dag Sverre From d.s.seljebotn at astro.uio.no Thu Mar 17 08:40:25 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 17 Mar 2011 08:40:25 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D81BA84.1020409@student.matnat.uio.no> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> Message-ID: <4D81BAE9.3090708@student.matnat.uio.no> On 03/17/2011 08:38 AM, Dag Sverre Seljebotn wrote: > On 03/17/2011 12:24 AM, Greg Ewing wrote: >> Stefan Behnel wrote: >> >>> I'm not sure if this is a good idea. "nogil" blocks don't have a way >>> to handle exceptions, so simply jumping out of them because an inner >>> 'with gil' block raised an exception can have unexpected side effects. >> >> Seems to me that the __Pyx_WriteUnraisable should be done at >> the end of the 'with gil' block, and execution should then >> continue from there. >> >> In other words, the effect on exception handling should be >> the same as if the 'with gil' block had been factored out into >> a separate function having no exception return value. >> > > -1. > > I consider the fact that exceptions don't propagate from some > functions a "currently unfixable bug". We should plan for it being > fixed some day. Having a "with" statement alter execution flow in this > way is totally unintuitive to me. > > If you want this, it's better to introduce a new keyword like > "trywithgil: ... except:" (not that I'm in favour of that). > > We could perhaps fix exception propagation from nogil functions by > using some conventions + setjmp/longjmp. Mono does this when calling > into native code, and I recently did it manually in Cython to > propagate exceptions through the Fortran wrappers in SciPy. Also, the > GIL may not be around forever even in CPython? (All arguments I've > seen for keeping it has been along the lines of "it slows down serial > code", not that it is considered a good thing.) Heh. I obviously meant that "removing it would slow down serial code". Dag > > Designing a language around the GIL feels like a dead-end to me. I'm > OK with being practical in the face of the limitations of today; but > let's keep "with gil" and "with nogil" something that can become noops > in the future without too much pain. Yes, I know that if the GIL goes > it will break Stefan's lxml code, and I'm sure other code -- I'm just > saying that we shouldn't make the language design even more > GIL-centric than it already is. > > Anyway, I'm off to write some computational Fortran+OpenMP code > because dealing with threading and Python is just more than I can deal > with... > > Dag Sverre > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Thu Mar 17 09:27:00 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 17 Mar 2011 09:27:00 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D81BA84.1020409@student.matnat.uio.no> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> Message-ID: <4D81C5D4.2070002@behnel.de> Dag Sverre Seljebotn, 17.03.2011 08:38: > On 03/17/2011 12:24 AM, Greg Ewing wrote: >> Stefan Behnel wrote: >> >>> I'm not sure if this is a good idea. "nogil" blocks don't have a way to >>> handle exceptions, so simply jumping out of them because an inner 'with >>> gil' block raised an exception can have unexpected side effects. >> >> Seems to me that the __Pyx_WriteUnraisable should be done at >> the end of the 'with gil' block, and execution should then >> continue from there. >> >> In other words, the effect on exception handling should be >> the same as if the 'with gil' block had been factored out into >> a separate function having no exception return value. >> > > -1. > > I consider the fact that exceptions don't propagate from some functions a > "currently unfixable bug". We should plan for it being fixed some day. It can't be fixed in general, because there are cases where exceptions simply cannot be propagated. Think of C callbacks, for example. C doesn't have a "normal" way of dealing with exceptions, so if an exception that originated from a callback simply leads to returning from the function, it may mean that the outer C code will simply continue to execute normally. Nothing's won in that case. In code: cdef void c_callback(...) nogil: ... do some C stuff ... with gil: ... do some Python stuff ... ... do some more C stuff ... So far, there are two proposed ways of doing this. 1) acquire the GIL on entry and exit, handling unraisable exceptions right before exiting. 2) keep all GIL requiring code inside of the "with gil" block, including unraisable exceptions. I find (2) a *lot* more intuitive, as well as much safer. We can't know what effects the surrounding "do C stuff" code has. It may contain thread-safe C level cleanup code for the "with gil" block, for example, or preparation code that enables returning into the calling C code. Simply jumping out of the GIL block without executing the trailing code may simply not work at all. > We could perhaps fix exception propagation from nogil functions by using > some conventions + setjmp/longjmp. Mono does this when calling into native > code, and I recently did it manually in Cython to propagate exceptions > through the Fortran wrappers in SciPy. Regardless of the topic of this thread, it would be nice to have longjmp support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work around several quirks in that area. > Also, the GIL may not be around > forever even in CPython? (All arguments I've seen for keeping it has been > along the lines of "it slows down serial code", not that it is considered a > good thing.) If it ever gets removed, there will surely have to be an emulation layer for C modules. Many of them simply use it as thread-lock, and that's totally reasonable IMHO. > Designing a language around the GIL feels like a dead-end to me. We keep having diverging opinions about the GIL. I like it, and I keep repeating myself by saying that "threading should be explicit". Having a way to lock the whole interpreter and to keep parallel execution and reentry points to well defined places in your code is a great feature. > I'm OK > with being practical in the face of the limitations of today; but let's > keep "with gil" and "with nogil" something that can become noops in the > future without too much pain. Yes, I know that if the GIL goes it will > break Stefan's lxml code, and I'm sure other code -- I'm just saying that > we shouldn't make the language design even more GIL-centric than it already > is. It's not. Even a removal of the GIL won't remove the fact that C can't propagate exceptions. Stefan From d.s.seljebotn at astro.uio.no Thu Mar 17 10:08:14 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 17 Mar 2011 10:08:14 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D81C5D4.2070002@behnel.de> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> Message-ID: <4D81CF7E.2090804@student.matnat.uio.no> On 03/17/2011 09:27 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 17.03.2011 08:38: >> On 03/17/2011 12:24 AM, Greg Ewing wrote: >>> Stefan Behnel wrote: >>> >>>> I'm not sure if this is a good idea. "nogil" blocks don't have a >>>> way to >>>> handle exceptions, so simply jumping out of them because an inner >>>> 'with >>>> gil' block raised an exception can have unexpected side effects. >>> >>> Seems to me that the __Pyx_WriteUnraisable should be done at >>> the end of the 'with gil' block, and execution should then >>> continue from there. >>> >>> In other words, the effect on exception handling should be >>> the same as if the 'with gil' block had been factored out into >>> a separate function having no exception return value. >>> >> >> -1. >> >> I consider the fact that exceptions don't propagate from some >> functions a >> "currently unfixable bug". We should plan for it being fixed some day. > > It can't be fixed in general, because there are cases where exceptions > simply cannot be propagated. Think of C callbacks, for example. C > doesn't have a "normal" way of dealing with exceptions, so if an > exception that originated from a callback simply leads to returning > from the function, it may mean that the outer C code will simply > continue to execute normally. Nothing's won in that case. Yes, that's a good point. (This is what I used setjmp/longjmp to work around BTW, to longjmp across the calling Fortran code. I knew it wasn't doing any mallocs/frees, let alone any file handling etc., so this was safe.) I'll admit that I'm mostly focused on code like def f(): with nogil: for ...: A if something_exceptional: with gil: raise Exception(...) B C where I'd say it's up to me to make sure that B and C can safely be skipped. It would be a major pain to have my raised exception here be "trapped" -- in fact, it would make the "with gil" statement unusable for my purposes. > > In code: > > cdef void c_callback(...) nogil: > ... do some C stuff ... > with gil: > ... do some Python stuff ... > ... do some more C stuff ... > > So far, there are two proposed ways of doing this. > > 1) acquire the GIL on entry and exit, handling unraisable exceptions > right before exiting. > > 2) keep all GIL requiring code inside of the "with gil" block, > including unraisable exceptions. > > I find (2) a *lot* more intuitive, as well as much safer. We can't > know what effects the surrounding "do C stuff" code has. It may > contain thread-safe C level cleanup code for the "with gil" block, for > example, or preparation code that enables returning into the calling C > code. Simply jumping out of the GIL block without executing the > trailing code may simply not work at all. I think you find (2) more intuitive because you have a very detailed knowledge of Cython and CPython, but that somebody new to Cython would expect a "with" statement to have the same control flow logic as the Python with statement. Of course, I don't have any data for that. How about this compromise: We balk on the code you wrote with: Error line 345: Exceptions propagating from "with gil" block cannot be propagated out of function, please insert try/except and handle exception So that we require this: with gil: try: ... except: warnings.warning(...) # or even cython.unraisable(e) This keeps me happy about not abusing the with statement for strange control flow, and makes the "with gil" useful for raising exceptions inside regular def functions with nogil blocks. > > >> We could perhaps fix exception propagation from nogil functions by using >> some conventions + setjmp/longjmp. Mono does this when calling into >> native >> code, and I recently did it manually in Cython to propagate exceptions >> through the Fortran wrappers in SciPy. > > Regardless of the topic of this thread, it would be nice to have > longjmp support in Cython. Lupa, my Cython wrapper for LuaJIT, > currently has to work around several quirks in that area. Not sure what you mean here, I used longjmp (in a function without any Python objects) and it seems to work just fine. Did I miss anything? > > >> Also, the GIL may not be around >> forever even in CPython? (All arguments I've seen for keeping it has >> been >> along the lines of "it slows down serial code", not that it is >> considered a >> good thing.) > > If it ever gets removed, there will surely have to be an emulation > layer for C modules. Many of them simply use it as thread-lock, and > that's totally reasonable IMHO. Good point. But there may be an option to disable said emulation layer that we want to make use of in Cython... (This is relevant today for Cython-on-.NET, for instance.) > > >> Designing a language around the GIL feels like a dead-end to me. > > We keep having diverging opinions about the GIL. I like it, and I keep > repeating myself by saying that "threading should be explicit". Having > a way to lock the whole interpreter and to keep parallel execution and > reentry points to well defined places in your code is a great feature. I think all of this just comes from using Cython for totally different things, which gives us different perspectives on the GIL. I guess we should just sit down in Munich and look at each other's codes and see if we can understand one another that way. A big discussion on whether "the GIL is good or not" is not very constructive. Dag Sverre From markflorisson88 at gmail.com Thu Mar 17 11:16:10 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 17 Mar 2011 11:16:10 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D81CF7E.2090804@student.matnat.uio.no> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> Message-ID: On 17 March 2011 10:08, Dag Sverre Seljebotn wrote: > On 03/17/2011 09:27 AM, Stefan Behnel wrote: >> >> Dag Sverre Seljebotn, 17.03.2011 08:38: >>> >>> On 03/17/2011 12:24 AM, Greg Ewing wrote: >>>> >>>> Stefan Behnel wrote: >>>> >>>>> I'm not sure if this is a good idea. "nogil" blocks don't have a way to >>>>> handle exceptions, so simply jumping out of them because an inner 'with >>>>> gil' block raised an exception can have unexpected side effects. >>>> >>>> Seems to me that the __Pyx_WriteUnraisable should be done at >>>> the end of the 'with gil' block, and execution should then >>>> continue from there. >>>> >>>> In other words, the effect on exception handling should be >>>> the same as if the 'with gil' block had been factored out into >>>> a separate function having no exception return value. >>>> >>> >>> -1. >>> >>> I consider the fact that exceptions don't propagate from some functions a >>> "currently unfixable bug". We should plan for it being fixed some day. >> >> It can't be fixed in general, because there are cases where exceptions >> simply cannot be propagated. Think of C callbacks, for example. C doesn't >> have a "normal" way of dealing with exceptions, so if an exception that >> originated from a callback simply leads to returning from the function, it >> may mean that the outer C code will simply continue to execute normally. >> Nothing's won in that case. > > Yes, that's a good point. (This is what I used setjmp/longjmp to work around > BTW, to longjmp across the calling Fortran code. I knew it wasn't doing any > mallocs/frees, let alone any file handling etc., so this was safe.) > > I'll admit that I'm mostly focused on code like > > def f(): > ? ?with nogil: > ? ? ? ?for ...: > ? ? ? ? ? ?A > ? ? ? ? ? ?if something_exceptional: > ? ? ? ? ? ? ? ?with gil: > ? ? ? ? ? ? ? ? ? ?raise Exception(...) > ? ? ? ? ? ?B > ? ? ? ?C > > where I'd say it's up to me to make sure that B and C can safely be skipped. > It would be a major pain to have my raised exception here be "trapped" -- in > fact, it would make the "with gil" statement unusable for my purposes. > > > >> >> In code: >> >> ? ?cdef void c_callback(...) nogil: >> ? ? ? ?... do some C stuff ... >> ? ? ? ?with gil: >> ? ? ? ? ? ?... do some Python stuff ... >> ? ? ? ?... do some more C stuff ... >> >> So far, there are two proposed ways of doing this. >> >> 1) acquire the GIL on entry and exit, handling unraisable exceptions right >> before exiting. >> >> 2) keep all GIL requiring code inside of the "with gil" block, including >> unraisable exceptions. >> >> I find (2) a *lot* more intuitive, as well as much safer. We can't know >> what effects the surrounding "do C stuff" code has. It may contain >> thread-safe C level cleanup code for the "with gil" block, for example, or >> preparation code that enables returning into the calling C code. Simply >> jumping out of the GIL block without executing the trailing code may simply >> not work at all. > > I think you find (2) more intuitive because you have a very detailed > knowledge of Cython and CPython, but that somebody new to Cython would > expect a "with" statement to have the same control flow logic as the Python > with statement. Of course, I don't have any data for that. > > How about this compromise: We balk on the code you wrote with: > > Error line 345: Exceptions propagating from "with gil" block cannot be > propagated out of function, please insert try/except and handle exception > > So that we require this: > > with gil: > ? ?try: > ? ? ? ?... > ? ?except: > ? ? ? ?warnings.warning(...) # or even cython.unraisable(e) > > This keeps me happy about not abusing the with statement for strange control > flow, and makes the "with gil" useful for raising exceptions inside regular > def functions with nogil blocks. > I agree with your previous statement, but not with your compromise :). We have to differentiate between two cases, similar to Stefan's cases, but different in a very important way that matter for nested GIL blocks. 1) Exceptions can propagate to some outer GIL section (in or outside the current function) 2) Exceptions can't propagate, because there is no outer GIL section and the function has a non-object return type With your compromise, with 1) exceptions cannot propagate, but with 2) you win forcing the user to be explicit. But then you still need to write to some variable indicating that an exception occurred and adjust control flow accordingly in your nogil section (unless you want to clean up and return immediately). If you have Python with-statement semantics, you can do the following, for instance: cdef void func() nogil: with gil: try: with nogil: with gil: code that may raise an exception this is not executed except ExceptionRaisedFromInnerWithGilBlock: handle exception here The point is, if you have case 2), and you want to use GIL code, you need to handle exceptions in some way. Forcing the user to not propagate anything doesn't sound right, unless this holds only for the outermost 'with gil' block. I would be OK with that, although it would be inconsistent with how exceptions in normal cdef functions with non-object return work, so I would say that we'd have to force it in the same manner there. >> >> >>> We could perhaps fix exception propagation from nogil functions by using >>> some conventions + setjmp/longjmp. Mono does this when calling into >>> native >>> code, and I recently did it manually in Cython to propagate exceptions >>> through the Fortran wrappers in SciPy. >> >> Regardless of the topic of this thread, it would be nice to have longjmp >> support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work >> around several quirks in that area. > > Not sure what you mean here, I used longjmp (in a function without any > Python objects) and it seems to work just fine. Did I miss anything? > >> >> >>> Also, the GIL may not be around >>> forever even in CPython? (All arguments I've seen for keeping it has been >>> along the lines of "it slows down serial code", not that it is considered >>> a >>> good thing.) >> >> If it ever gets removed, there will surely have to be an emulation layer >> for C modules. Many of them simply use it as thread-lock, and that's totally >> reasonable IMHO. > > Good point. But there may be an option to disable said emulation layer that > we want to make use of in Cython... > > (This is relevant today for Cython-on-.NET, for instance.) > >> >> >>> Designing a language around the GIL feels like a dead-end to me. >> >> We keep having diverging opinions about the GIL. I like it, and I keep >> repeating myself by saying that "threading should be explicit". Having a way >> to lock the whole interpreter and to keep parallel execution and reentry >> points to well defined places in your code is a great feature. > > I think all of this just comes from using Cython for totally different > things, which gives us different perspectives on the GIL. I guess we should > just sit down in Munich and look at each other's codes and see if we can > understand one another that way. A big discussion on whether "the GIL is > good or not" is not very constructive. > > Dag Sverre > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Thu Mar 17 11:18:04 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 17 Mar 2011 11:18:04 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D81C5D4.2070002@behnel.de> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> Message-ID: On 17 March 2011 09:27, Stefan Behnel wrote: > Dag Sverre Seljebotn, 17.03.2011 08:38: >> >> On 03/17/2011 12:24 AM, Greg Ewing wrote: >>> >>> Stefan Behnel wrote: >>> >>>> I'm not sure if this is a good idea. "nogil" blocks don't have a way to >>>> handle exceptions, so simply jumping out of them because an inner 'with >>>> gil' block raised an exception can have unexpected side effects. >>> >>> Seems to me that the __Pyx_WriteUnraisable should be done at >>> the end of the 'with gil' block, and execution should then >>> continue from there. >>> >>> In other words, the effect on exception handling should be >>> the same as if the 'with gil' block had been factored out into >>> a separate function having no exception return value. >>> >> >> -1. >> >> I consider the fact that exceptions don't propagate from some functions a >> "currently unfixable bug". We should plan for it being fixed some day. > > It can't be fixed in general, because there are cases where exceptions > simply cannot be propagated. Think of C callbacks, for example. C doesn't > have a "normal" way of dealing with exceptions, so if an exception that > originated from a callback simply leads to returning from the function, it > may mean that the outer C code will simply continue to execute normally. > Nothing's won in that case. > > In code: > > ? ?cdef void c_callback(...) nogil: > ? ? ? ?... do some C stuff ... > ? ? ? ?with gil: > ? ? ? ? ? ?... do some Python stuff ... > ? ? ? ?... do some more C stuff ... > > So far, there are two proposed ways of doing this. > > 1) acquire the GIL on entry and exit, handling unraisable exceptions right > before exiting. > > 2) keep all GIL requiring code inside of the "with gil" block, including > unraisable exceptions. > > I find (2) a *lot* more intuitive, as well as much safer. We can't know what > effects the surrounding "do C stuff" code has. It may contain thread-safe C > level cleanup code for the "with gil" block, for example, or preparation > code that enables returning into the calling C code. Simply jumping out of > the GIL block without executing the trailing code may simply not work at > all. Which is exactly why users will have to handle exceptions if they want their C code to execute. > >> We could perhaps fix exception propagation from nogil functions by using >> some conventions + setjmp/longjmp. Mono does this when calling into native >> code, and I recently did it manually in Cython to propagate exceptions >> through the Fortran wrappers in SciPy. > > Regardless of the topic of this thread, it would be nice to have longjmp > support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work > around several quirks in that area. > > >> Also, the GIL may not be around >> forever even in CPython? (All arguments I've seen for keeping it has been >> along the lines of "it slows down serial code", not that it is considered >> a >> good thing.) > > If it ever gets removed, there will surely have to be an emulation layer for > C modules. Many of them simply use it as thread-lock, and that's totally > reasonable IMHO. > > >> Designing a language around the GIL feels like a dead-end to me. > > We keep having diverging opinions about the GIL. I like it, and I keep > repeating myself by saying that "threading should be explicit". Having a way > to lock the whole interpreter and to keep parallel execution and reentry > points to well defined places in your code is a great feature. > > >> I'm OK >> with being practical in the face of the limitations of today; but let's >> keep "with gil" and "with nogil" something that can become noops in the >> future without too much pain. Yes, I know that if the GIL goes it will >> break Stefan's lxml code, and I'm sure other code -- I'm just saying that >> we shouldn't make the language design even more GIL-centric than it >> already >> is. > > It's not. Even a removal of the GIL won't remove the fact that C can't > propagate exceptions. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From d.s.seljebotn at astro.uio.no Thu Mar 17 11:35:19 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 17 Mar 2011 11:35:19 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> Message-ID: <4D81E3E7.4030309@student.matnat.uio.no> On 03/17/2011 11:16 AM, mark florisson wrote: > On 17 March 2011 10:08, Dag Sverre Seljebotn wrote: >> >> How about this compromise: We balk on the code you wrote with: >> >> Error line 345: Exceptions propagating from "with gil" block cannot be >> propagated out of function, please insert try/except and handle exception >> >> So that we require this: >> >> with gil: >> try: >> ... >> except: >> warnings.warning(...) # or even cython.unraisable(e) >> >> This keeps me happy about not abusing the with statement for strange control >> flow, and makes the "with gil" useful for raising exceptions inside regular >> def functions with nogil blocks. >> > I agree with your previous statement, but not with your compromise :). > We have to differentiate between two cases, similar to Stefan's cases, > but different in a very important way that matter for nested GIL > blocks. > > 1) Exceptions can propagate to some outer GIL section (in or outside > the current function) > 2) Exceptions can't propagate, because there is no outer GIL section > and the function has a non-object return type > > With your compromise, with 1) exceptions cannot propagate, but with 2) > you win forcing the user to be explicit. But then you still need to > write to some variable indicating that an exception occurred and > adjust control flow accordingly in your nogil section (unless you want > to clean up and return immediately). > > If you have Python with-statement semantics, you can do the following, > for instance: > > cdef void func() nogil: > with gil: > try: > > with nogil: > with gil: > code that may raise an exception > > this is not executed > > except ExceptionRaisedFromInnerWithGilBlock: > handle exception here > > The point is, if you have case 2), and you want to use GIL code, you > need to handle exceptions in some way. Forcing the user to not > propagate anything doesn't sound right, unless this holds only for the > outermost 'with gil' block. I would be OK with that, although it would > be inconsistent with how exceptions in normal cdef functions with > non-object return work, so I would say that we'd have to force it in > the same manner there. I think we should perhaps look at enforcing explicit exception-ignoring everywhere.. there's a lot of details to hash out, and there's the issue of backwards compatability, but it could be dealt with with a couple of releases where we only raise a warning and so on. It could involve a *very* limited subset of exception handling for use in nogil mode (i.e., only a bare "except:" statement allowed, where one can call either "cython.unraisable()", "pass", or set a flag). Dag Sverre From markflorisson88 at gmail.com Thu Mar 17 11:43:39 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 17 Mar 2011 11:43:39 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D81E3E7.4030309@student.matnat.uio.no> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D81E3E7.4030309@student.matnat.uio.no> Message-ID: On 17 March 2011 11:35, Dag Sverre Seljebotn wrote: > On 03/17/2011 11:16 AM, mark florisson wrote: >> >> On 17 March 2011 10:08, Dag Sverre Seljebotn >> ?wrote: >>> >>> How about this compromise: We balk on the code you wrote with: >>> >>> Error line 345: Exceptions propagating from "with gil" block cannot be >>> propagated out of function, please insert try/except and handle exception >>> >>> So that we require this: >>> >>> with gil: >>> ? ?try: >>> ? ? ? ?... >>> ? ?except: >>> ? ? ? ?warnings.warning(...) # or even cython.unraisable(e) >>> >>> This keeps me happy about not abusing the with statement for strange >>> control >>> flow, and makes the "with gil" useful for raising exceptions inside >>> regular >>> def functions with nogil blocks. >>> >> I agree with your previous statement, but not with your compromise :). >> We have to differentiate between two cases, similar to Stefan's cases, >> but different in a very important way that matter for nested GIL >> blocks. >> >> 1) Exceptions can propagate to some outer GIL section (in or outside >> the current function) >> 2) Exceptions can't propagate, because there is no outer GIL section >> and the function has a non-object return type >> >> With your compromise, with 1) exceptions cannot propagate, but with 2) >> you win forcing the user to be explicit. But then you still need to >> write to some variable indicating that an exception occurred and >> adjust control flow accordingly in your nogil section (unless you want >> to clean up and return immediately). >> >> If you have Python with-statement semantics, you can do the following, >> for instance: >> >> cdef void func() nogil: >> ? ? with gil: >> ? ? ? ? try: >> >> ? ? ? ? ? ? with nogil: >> ? ? ? ? ? ? ? ? with gil: >> ? ? ? ? ? ? ? ? ? ? code that may raise an exception >> >> ? ? ? ? ? ? ? ? this is not executed >> >> ? ? ? ? except ExceptionRaisedFromInnerWithGilBlock: >> ? ? ? ? ? ? handle exception here >> >> The point is, if you have case 2), and you want to use GIL code, you >> need to handle exceptions in some way. Forcing the user to not >> propagate anything doesn't sound right, unless this holds only for the >> outermost 'with gil' block. I would be OK with that, although it would >> be inconsistent with how exceptions in normal cdef functions with >> non-object return work, so I would say that we'd have to force it in >> the same manner there. > > I think we should perhaps look at enforcing explicit exception-ignoring > everywhere.. there's a lot of details to hash out, and there's the issue of > backwards compatability, but it could be dealt with with a couple of > releases where we only raise a warning and so on. > > It could involve a *very* limited subset of exception handling for use in > nogil mode (i.e., only a bare "except:" statement allowed, where one can > call either "cython.unraisable()", "pass", or set a flag). I don't think we should allow the forced exception handling in nogil sections, but in gil sections (and you'd have to hold the GIL as exceptions are stored on the thread state). So forced exception handling for functions declared 'with gil' and for outermost 'with gil' blocks in cdef functions with non-object return. > Dag Sverre > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From jason-sage at creativetrax.com Thu Mar 17 14:15:26 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Thu, 17 Mar 2011 08:15:26 -0500 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> Message-ID: <4D82096E.4090605@creativetrax.com> On 3/16/11 11:05 PM, Robert Bradshaw wrote: > On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: >> I'm actually leaning towards not guaranteeing the order of execution if C >> doesn't do it either. If this is really required, it's easy to work around >> for users, but it's severely hard to fix for Cython in all cases, and the >> gain is truly small. After all, we'd only make it easier for users to write >> bad code. > > Yep. Lets keep the code in for the above case. Is there a huge big warning in the docs? Maybe on this page would be a good place: http://docs.cython.org/src/userguide/limitations.html Jason From robertwb at math.washington.edu Thu Mar 17 21:37:58 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 17 Mar 2011 13:37:58 -0700 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D82096E.4090605@creativetrax.com> References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> <4D82096E.4090605@creativetrax.com> Message-ID: On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout wrote: > On 3/16/11 11:05 PM, Robert Bradshaw wrote: >> >> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel >> ?wrote: >>> >>> I'm actually leaning towards not guaranteeing the order of execution if C >>> doesn't do it either. If this is really required, it's easy to work >>> around >>> for users, but it's severely hard to fix for Cython in all cases, and the >>> gain is truly small. After all, we'd only make it easier for users to >>> write >>> bad code. >> >> Yep. Lets keep the code in for the above case. > > > Is there a huge big warning in the docs? ?Maybe on this page would be a good > place: http://docs.cython.org/src/userguide/limitations.html This doesn't affect Python functions at all, so I'm not sure it belongs on that page. I agree that it should be mentioned that c(p)def functions have C calling semantics, *including* an unspecified order or argument evaluation. - Robert From vitja.makarov at gmail.com Thu Mar 17 21:48:10 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 17 Mar 2011 23:48:10 +0300 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> <4D82096E.4090605@creativetrax.com> Message-ID: 2011/3/17 Robert Bradshaw : > On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout > wrote: >> On 3/16/11 11:05 PM, Robert Bradshaw wrote: >>> >>> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel >>> ?wrote: >>>> >>>> I'm actually leaning towards not guaranteeing the order of execution if C >>>> doesn't do it either. If this is really required, it's easy to work >>>> around >>>> for users, but it's severely hard to fix for Cython in all cases, and the >>>> gain is truly small. After all, we'd only make it easier for users to >>>> write >>>> bad code. >>> >>> Yep. Lets keep the code in for the above case. >> >> >> Is there a huge big warning in the docs? ?Maybe on this page would be a good >> place: http://docs.cython.org/src/userguide/limitations.html > > This doesn't affect Python functions at all, so I'm not sure it > belongs on that page. I agree that it should be mentioned that c(p)def > functions have C calling semantics, *including* an unspecified order > or argument evaluation. > As you noticed above, how should be handled def function inlining? I guess there are restrictions on def function argument type, so side-effect isn't issue here. -- vitja. From robertwb at math.washington.edu Thu Mar 17 21:57:06 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 17 Mar 2011 13:57:06 -0700 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> <4D82096E.4090605@creativetrax.com> Message-ID: On Thu, Mar 17, 2011 at 1:48 PM, Vitja Makarov wrote: > 2011/3/17 Robert Bradshaw : >> On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout >> wrote: >>> On 3/16/11 11:05 PM, Robert Bradshaw wrote: >>>> >>>> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel >>>> ?wrote: >>>>> >>>>> I'm actually leaning towards not guaranteeing the order of execution if C >>>>> doesn't do it either. If this is really required, it's easy to work >>>>> around >>>>> for users, but it's severely hard to fix for Cython in all cases, and the >>>>> gain is truly small. After all, we'd only make it easier for users to >>>>> write >>>>> bad code. >>>> >>>> Yep. Lets keep the code in for the above case. >>> >>> >>> Is there a huge big warning in the docs? ?Maybe on this page would be a good >>> place: http://docs.cython.org/src/userguide/limitations.html >> >> This doesn't affect Python functions at all, so I'm not sure it >> belongs on that page. I agree that it should be mentioned that c(p)def >> functions have C calling semantics, *including* an unspecified order >> or argument evaluation. >> > > As you noticed above, how should be handled def function inlining? > > I guess there are restrictions on def function argument type, so > side-effect isn't issue here. Yep, or at least not near as much of an issue. (I think the C compiler can optimize away an, e.g, extra copy of, e.g, int, double and PyObject* temps in most cases.) - Robert From stefan_ml at behnel.de Thu Mar 17 22:25:43 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 17 Mar 2011 22:25:43 +0100 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> Message-ID: <4D827C57.5010905@behnel.de> Robert Bradshaw, 17.03.2011 05:05: > On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: >> I'm actually leaning towards not guaranteeing the order of execution if C >> doesn't do it either. If this is really required, it's easy to work around >> for users, but it's severely hard to fix for Cython in all cases, and the >> gain is truly small. After all, we'd only make it easier for users to write >> bad code. > > Yep. Lets keep the code in for the above case. Erm, the current status is that we try to guarantee the order by pushing everything into temps, thus breaking Sage. What code exactly did you intend to keep here? Stefan From robertwb at math.washington.edu Thu Mar 17 22:26:43 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 17 Mar 2011 14:26:43 -0700 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: <4D827C57.5010905@behnel.de> References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> <4D827C57.5010905@behnel.de> Message-ID: On Thu, Mar 17, 2011 at 2:25 PM, Stefan Behnel wrote: > Robert Bradshaw, 17.03.2011 05:05: >> >> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: >>> >>> I'm actually leaning towards not guaranteeing the order of execution if C >>> doesn't do it either. If this is really required, it's easy to work >>> around >>> for users, but it's severely hard to fix for Cython in all cases, and the >>> gain is truly small. After all, we'd only make it easier for users to >>> write >>> bad code. >> >> Yep. Lets keep the code in for the above case. > > Erm, the current status is that we try to guarantee the order by pushing > everything into temps, thus breaking Sage. What code exactly did you intend > to keep here? I was thinking about guarding it with an if False (or flag in Options.py). - Robert From stefan_ml at behnel.de Thu Mar 17 22:35:41 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 17 Mar 2011 22:35:41 +0100 Subject: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654) In-Reply-To: References: <4D7A1957.6070107@behnel.de> <4D7A2CCD.6060604@behnel.de> <4D7A5DAF.1020003@behnel.de> <4D80CEEF.4080701@behnel.de> <4D827C57.5010905@behnel.de> Message-ID: <4D827EAD.8070005@behnel.de> Robert Bradshaw, 17.03.2011 22:26: > On Thu, Mar 17, 2011 at 2:25 PM, Stefan Behnel wrote: >> Robert Bradshaw, 17.03.2011 05:05: >>> >>> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: >>>> >>>> I'm actually leaning towards not guaranteeing the order of execution if C >>>> doesn't do it either. If this is really required, it's easy to work >>>> around >>>> for users, but it's severely hard to fix for Cython in all cases, and the >>>> gain is truly small. After all, we'd only make it easier for users to >>>> write >>>> bad code. >>> >>> Yep. Lets keep the code in for the above case. >> >> Erm, the current status is that we try to guarantee the order by pushing >> everything into temps, thus breaking Sage. What code exactly did you intend >> to keep here? > > I was thinking about guarding it with an if False (or flag in Options.py). Ah - the "poor man's VCS" approach? ;) Stefan From greg.ewing at canterbury.ac.nz Fri Mar 18 00:32:55 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 Mar 2011 12:32:55 +1300 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D81CF7E.2090804@student.matnat.uio.no> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> Message-ID: <4D829A27.60405@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > def f(): > with nogil: > for ...: > A > if something_exceptional: > with gil: > raise Exception(...) > B > C If that's to be supported, the following really ought to be supported as well: def f(): with nogil: try: ... with gil: raise Exception() finally: ...do some cleanup... -- Greg From markflorisson88 at gmail.com Fri Mar 18 00:49:04 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 18 Mar 2011 00:49:04 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D829A27.60405@canterbury.ac.nz> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> Message-ID: On 18 March 2011 00:32, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> def f(): >> ? ?with nogil: >> ? ? ? ?for ...: >> ? ? ? ? ? ?A >> ? ? ? ? ? ?if something_exceptional: >> ? ? ? ? ? ? ? ?with gil: >> ? ? ? ? ? ? ? ? ? ?raise Exception(...) >> ? ? ? ? ? ?B >> ? ? ? ?C > > If that's to be supported, the following really ought to be > supported as well: > > ?def f(): > ? ? with nogil: > ? ? ? ?try: > ? ? ? ? ? ... > ? ? ? ? ? with gil: > ? ? ? ? ? ? ?raise Exception() > ? ? ? ?finally: > ? ? ? ? ? ...do some cleanup... > Why? I assume in his example the for loop was a Cython C for loop, not one that deals with Python objects. If you want to do cleanup you can catch the exception in the 'with gil:' block, or use try/finally in the 'with gil:' block or outside the 'with nogil:' block. Special-casing try/finally in nogil sections for only 'with gil:' sounds somewhat weird. On the other hand, it may be somewhat more convenient, and I think we could support it without having to acquire the GIL in the finally clause. I wouldn't be particularly opposed to that. > Greg > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Fri Mar 18 00:56:18 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 18 Mar 2011 00:56:18 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D80D3D1.6000601@behnel.de> Message-ID: On 16 March 2011 20:16, mark florisson wrote: >>>> Could someone review the patch (which is attached)? Maybe check if I >>>> haven't missed any side cases and such? >>> >>> From a first look, the test file you added seems far too short. I would >>> expect that this feature requires a lot more testing in combination with >>> declared and undeclared local variables, type inference, several exception >>> raising and catching situations (e.g. raise in one block, catch in an outer >>> block, try-finally, ...) or looping. It may also have an impact on Vitja's >>> control flow analysis branch that's worth considering. >> >> I agree. I think I'll start a branch and work on some more tests. > > I also have to adjust the function teardown locking. I'll start a > branch and report back if I get positive results. > I added more tests, the code can be found in this fork: https://github.com/markflorisson88/cython . There is currently no compile-time checking for exceptions that might be swallowed, it still works in the same way as normal cdef functions with a non-object return type. Should we issue warnings for such cases instead of relying on the implicit swallow-and-print, as Dag suggested? From greg.ewing at canterbury.ac.nz Fri Mar 18 01:18:58 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 18 Mar 2011 13:18:58 +1300 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> Message-ID: <4D82A4F2.5080200@canterbury.ac.nz> mark florisson wrote: > I think we could support it without having to acquire > the GIL in the finally clause. That was the intention -- the code in the finally clause would be subject to the same nogil restrictions as the rest of the nogil block. My point is that as long as you're allowing exceptions to be tunnelled through nogil blocks, they should respect any finally clauses that they pass through on the way. -- Greg From stefan_ml at behnel.de Fri Mar 18 07:07:31 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Mar 2011 07:07:31 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D82A4F2.5080200@canterbury.ac.nz> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> <4D82A4F2.5080200@canterbury.ac.nz> Message-ID: <4D82F6A3.4020600@behnel.de> Greg Ewing, 18.03.2011 01:18: > mark florisson wrote: >> I think we could support it without having to acquire >> the GIL in the finally clause. > > That was the intention -- the code in the finally clause would > be subject to the same nogil restrictions as the rest of > the nogil block. > > My point is that as long as you're allowing exceptions to be > tunnelled through nogil blocks, they should respect any finally > clauses that they pass through on the way. +1 Stefan From markflorisson88 at gmail.com Fri Mar 18 10:52:16 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 18 Mar 2011 10:52:16 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D82F6A3.4020600@behnel.de> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> <4D82A4F2.5080200@canterbury.ac.nz> <4D82F6A3.4020600@behnel.de> Message-ID: On 18 March 2011 07:07, Stefan Behnel wrote: > Greg Ewing, 18.03.2011 01:18: >> >> mark florisson wrote: >>> >>> I think we could support it without having to acquire >>> the GIL in the finally clause. >> >> That was the intention -- the code in the finally clause would >> be subject to the same nogil restrictions as the rest of >> the nogil block. >> >> My point is that as long as you're allowing exceptions to be >> tunnelled through nogil blocks, they should respect any finally >> clauses that they pass through on the way. > > +1 Ok, I will give it a go and try to allow it when they surround with gil blocks. I would however like to reiterate that it is a special-case, inconsistent with previous behaviour, and basically extends the language and won't work for functions that are called and declared 'with gil'. But it is convenient, so I can't help but like it at the same time :] > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Fri Mar 18 11:10:46 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Mar 2011 11:10:46 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> <4D82A4F2.5080200@canterbury.ac.nz> <4D82F6A3.4020600@behnel.de> Message-ID: <4D832FA6.4020301@behnel.de> mark florisson, 18.03.2011 10:52: > On 18 March 2011 07:07, Stefan Behnel wrote: >> Greg Ewing, 18.03.2011 01:18: >>> >>> mark florisson wrote: >>>> >>>> I think we could support it without having to acquire >>>> the GIL in the finally clause. >>> >>> That was the intention -- the code in the finally clause would >>> be subject to the same nogil restrictions as the rest of >>> the nogil block. >>> >>> My point is that as long as you're allowing exceptions to be >>> tunnelled through nogil blocks, they should respect any finally >>> clauses that they pass through on the way. >> >> +1 > > Ok, I will give it a go and try to allow it when they surround with > gil blocks. I would however like to reiterate that it is a > special-case, inconsistent with previous behaviour, and basically > extends the language and won't work for functions that are called and > declared 'with gil'. But it is convenient, so I can't help but like it > at the same time :] I'm not sure I understand why you think it's so bad, and why it would be inconsistent with previous behaviour. The only real problem I see is that you could do things like this: with nogil: try: with gil: raise ... finally: with gil: raise ... i.e. you could loose the original exception. Even worse: with nogil: try: with gil: raise ... finally: with gil: try: raise except: pass Here, it must be made sure that the original exception still gets raised properly at the end. That's a problem in Py2, where exceptions are badly scoped, i.e. Python code that runs in the interpreter could fail to reset the original exception after catching another one. But I guess these things are up to the "with gil" block/function, rather than the above "finally" clause. Actually, I think I still find it more convenient to not provide *any* special exception paths through nogil code, i.e. to not let exceptions in "with gil" blocks exit from outer "nogil" blocks. That would avoid all of the semantic difficulties above. Stefan From d.s.seljebotn at astro.uio.no Fri Mar 18 12:07:54 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 18 Mar 2011 12:07:54 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D832FA6.4020301@behnel.de> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> <4D82A4F2.5080200@canterbury.ac.nz> <4D82F6A3.4020600@behnel.de> <4D832FA6.4020301@behnel.de> Message-ID: <4D833D0A.2090202@student.matnat.uio.no> On 03/18/2011 11:10 AM, Stefan Behnel wrote: > mark florisson, 18.03.2011 10:52: >> On 18 March 2011 07:07, Stefan Behnel wrote: >>> Greg Ewing, 18.03.2011 01:18: >>>> >>>> mark florisson wrote: >>>>> >>>>> I think we could support it without having to acquire >>>>> the GIL in the finally clause. >>>> >>>> That was the intention -- the code in the finally clause would >>>> be subject to the same nogil restrictions as the rest of >>>> the nogil block. >>>> >>>> My point is that as long as you're allowing exceptions to be >>>> tunnelled through nogil blocks, they should respect any finally >>>> clauses that they pass through on the way. >>> >>> +1 >> >> Ok, I will give it a go and try to allow it when they surround with >> gil blocks. I would however like to reiterate that it is a >> special-case, inconsistent with previous behaviour, and basically >> extends the language and won't work for functions that are called and >> declared 'with gil'. But it is convenient, so I can't help but like it >> at the same time :] > > I'm not sure I understand why you think it's so bad, and why it would > be inconsistent with previous behaviour. > > The only real problem I see is that you could do things like this: > > with nogil: > try: > with gil: raise ... > finally: > with gil: raise ... > > i.e. you could loose the original exception. Even worse: > > with nogil: > try: > with gil: raise ... > finally: > with gil: > try: raise > except: pass > > Here, it must be made sure that the original exception still gets > raised properly at the end. That's a problem in Py2, where exceptions > are badly scoped, i.e. Python code that runs in the interpreter could > fail to reset the original exception after catching another one. But I > guess these things are up to the "with gil" block/function, rather > than the above "finally" clause. > > Actually, I think I still find it more convenient to not provide *any* > special exception paths through nogil code, i.e. to not let exceptions > in "with gil" blocks exit from outer "nogil" blocks. That would avoid > all of the semantic difficulties above. Well, of course not supporting something is easier. But is it user friendly? Relying on boolean flags to signal errors states is a real pain when one is used to using exceptions. Dag Sverre From stefan_ml at behnel.de Fri Mar 18 13:36:01 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Mar 2011 13:36:01 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D833D0A.2090202@student.matnat.uio.no> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> <4D82A4F2.5080200@canterbury.ac.nz> <4D82F6A3.4020600@behnel.de> <4D832FA6.4020301@behnel.de> <4D833D0A.2090202@student.matnat.uio.no> Message-ID: <4D8351B1.8090006@behnel.de> Dag Sverre Seljebotn, 18.03.2011 12:07: > On 03/18/2011 11:10 AM, Stefan Behnel wrote: >> Actually, I think I still find it more convenient to not provide *any* >> special exception paths through nogil code, i.e. to not let exceptions in >> "with gil" blocks exit from outer "nogil" blocks. That would avoid all of >> the semantic difficulties above. > > Well, of course not supporting something is easier. That's a bit exaggerated, IMHO. I would expect that there'd be many cases where the right thing to do in case of an exception is to return from a function, with a bit of explicit cleanup. So many exceptions won't leave the "with gil" block anyways. And few users will actually use nested "with gil" and "nogil" blocks. But still, there's the question how code that re-acquires the GIL should behave (still thinking of a callback case here). Should it check for a raised exception and jump into the exception handling path? Or should it ignore it and continue until something tests for the exception explicitly? And what if intermediately running nogil code changed some state somewhere that breaks the exception handling path? Or what if the code block that temporarily re-acquires the GIL (e.g. a logging handler) is completely unrelated to the one that actually raised the exception or the one that called into the whole machinery (and that would be the right one to handle it)? These things are not trivial, neither for Cython's language semantics nor for users. We shouldn't forget that basically all Python operations can at least raise a MemoryError or a KeyboardInterrupt etc. Most users won't think of these cases. I think it would help users in writing safer code if the need to handle exceptions in nogil blocks was always made explicit by the compiler. That's different from "not supporting something". > Relying on boolean flags to signal errors states is a real pain > when one is used to using exceptions. Well, if those exception do not take safe and obvious paths, it may still be better to use explicit boolean flags instead. Specifically, I'm fine with letting exceptions flow through nogil code iff there is a way to react on exceptional exit paths inside of nogil blocks. I'm against enabling "except" clauses in nogil blocks, but I think a try-finally would catch the most important use cases iff we can figure out clean semantics for this, including the cases I mentioned previously. That's the condition I see. Stefan From markflorisson88 at gmail.com Fri Mar 18 14:31:16 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 18 Mar 2011 14:31:16 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D8351B1.8090006@behnel.de> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> <4D82A4F2.5080200@canterbury.ac.nz> <4D82F6A3.4020600@behnel.de> <4D832FA6.4020301@behnel.de> <4D833D0A.2090202@student.matnat.uio.no> <4D8351B1.8090006@behnel.de> Message-ID: On 18 March 2011 13:36, Stefan Behnel wrote: > Dag Sverre Seljebotn, 18.03.2011 12:07: >> >> On 03/18/2011 11:10 AM, Stefan Behnel wrote: >>> >>> Actually, I think I still find it more convenient to not provide *any* >>> special exception paths through nogil code, i.e. to not let exceptions in >>> "with gil" blocks exit from outer "nogil" blocks. That would avoid all of >>> the semantic difficulties above. >> >> Well, of course not supporting something is easier. > > That's a bit exaggerated, IMHO. I would expect that there'd be many cases > where the right thing to do in case of an exception is to return from a > function, with a bit of explicit cleanup. So many exceptions won't leave the > "with gil" block anyways. And few users will actually use nested "with gil" > and "nogil" blocks. This also holds for cdef nogil functions with 'with gil' blocks. > But still, there's the question how code that re-acquires the GIL should > behave (still thinking of a callback case here). Should it check for a > raised exception and jump into the exception handling path? Or should it > ignore it and continue until something tests for the exception explicitly? I'm not sure what you mean, are you talking about exceptions raised in the with gil block, or about exceptions that were pending before the function was called? In the latter, case, my answer would be, "in the same way that functions declared 'with gil' work": you assume there is no pending exception. It doesn't make sense to call Python-related code with an exception set. And indeed, the finally clause works by first clearing and storing the exception on the stack, and then restoring it afterwards if no other exception propagated. This is not a case we should strife to handle, it's incorrect code. > And what if intermediately running nogil code changed some state somewhere > that breaks the exception handling path? If users call into C code that raises an exception, it is their task to take care of any exceptions manually, or by using the right Cython declarations (such as the 'except' declaration clause for cdef functions). This is not any different from any other semantics we currently have for gil sections. > Or what if the code block that > temporarily re-acquires the GIL (e.g. a logging handler) is completely > unrelated to the one that actually raised the exception or the one that > called into the whole machinery (and that would be the right one to handle > it)? These things are not trivial, neither for Cython's language semantics > nor for users. If this happens in a with gil section in the finally clause of a nogil section, then I think they should be chained (in python 3). In python 2 the exception should simply be replaced, as per the semantics of finally. > We shouldn't forget that basically all Python operations can at least raise > a MemoryError or a KeyboardInterrupt etc. Most users won't think of these > cases. I think it would help users in writing safer code if the need to > handle exceptions in nogil blocks was always made explicit by the compiler. > That's different from "not supporting something". > >> Relying on boolean flags to signal errors states is a real pain >> when one is used to using exceptions. > > Well, if those exception do not take safe and obvious paths, it may still be > better to use explicit boolean flags instead. Again, I think "Python semantics" are quite obvious for exception paths from with gil statements. If the user really needs to do cleanup, exceptions may be caught in the with gil block and an error flag used, or the cleanup can be moved to the finally clause in the gil section. Without compiler enforcement, the user will forget the try/finally around the gil section as easily as the try/except or try/finally around the body of the with gil section. I don't find enforcing it useful at all, though. The only thing that would be useful is enforcing the user to handle exceptions when they might be unraisable due to the function's return type. > Specifically, I'm fine with letting exceptions flow through nogil code iff > there is a way to react on exceptional exit paths inside of nogil blocks. > I'm against enabling "except" clauses in nogil blocks, but I think a > try-finally would catch the most important use cases iff we can figure out > clean semantics for this, including the cases I mentioned previously. That's > the condition I see. To summarize, I think the semantics should be like this: - propagate exceptions from with gil blocks through nogil sections - if there is a finally clause in a nogil section corresponding to a try surrounding a with gil block, execute it - if the finally clause has with gil blocks, allow them to raise exceptions, thereby overriding any pending exception, using the semantics just mentioned (applied recursively) > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Fri Mar 18 15:05:18 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 18 Mar 2011 15:05:18 +0100 Subject: [Cython] 'with gil:' statement In-Reply-To: <4D8351B1.8090006@behnel.de> References: <4D80D3D1.6000601@behnel.de> <4D8146CB.6060608@canterbury.ac.nz> <4D81BA84.1020409@student.matnat.uio.no> <4D81C5D4.2070002@behnel.de> <4D81CF7E.2090804@student.matnat.uio.no> <4D829A27.60405@canterbury.ac.nz> <4D82A4F2.5080200@canterbury.ac.nz> <4D82F6A3.4020600@behnel.de> <4D832FA6.4020301@behnel.de> <4D833D0A.2090202@student.matnat.uio.no> <4D8351B1.8090006@behnel.de> Message-ID: <4D83669E.9030205@behnel.de> Stefan Behnel, 18.03.2011 13:36: > We shouldn't forget that basically all Python operations can at least raise > a MemoryError or a KeyboardInterrupt etc. Most users won't think of these > cases. I think it would help users in writing safer code if the need to > handle exceptions in nogil blocks was always made explicit by the compiler. Oh, and another nice thing to add here: A bare "except" clause can actually raise an exception if extracting/instantiating the exception fails. So there's really no way for user code to catch all exceptions and there will always be unraisable exceptions in code that is technically incapable of propagating them. Stefan From vitja.makarov at gmail.com Mon Mar 21 11:45:02 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 21 Mar 2011 13:45:02 +0300 Subject: [Cython] Message system refactoring Message-ID: Now error/warning messages are stored in global variables at Cython.Compiler.Errors I think it's much better to move error handling into some object, Main.Context for example. Some benefits: - reduce use of global variables - allow more then one cython compiler instance at the time - make it much easy to implement -Werror - cython directives can affect message system (fast_fail, werror) - messages could be easily sorted -- vitja. From d.s.seljebotn at astro.uio.no Mon Mar 21 11:51:10 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 21 Mar 2011 11:51:10 +0100 Subject: [Cython] Message system refactoring In-Reply-To: References: Message-ID: <4D872D9E.4000905@student.matnat.uio.no> On 03/21/2011 11:45 AM, Vitja Makarov wrote: > Now error/warning messages are stored in global variables at > Cython.Compiler.Errors > > I think it's much better to move error handling into some object, > Main.Context for example. > > Some benefits: > - reduce use of global variables > - allow more then one cython compiler instance at the time > - make it much easy to implement -Werror > - cython directives can affect message system (fast_fail, werror) > - messages could be easily sorted +1. I assume the reason this is not done is simply because it would be a lot of work and the payback is less than spending time on other stuff. By attaching the error context to "env" and "code" one can avoid a lot of signature changes. I think transforms should take the context in their constructor. Dag Sverre From vitja.makarov at gmail.com Mon Mar 21 12:01:11 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 21 Mar 2011 14:01:11 +0300 Subject: [Cython] Message system refactoring In-Reply-To: <4D872D9E.4000905@student.matnat.uio.no> References: <4D872D9E.4000905@student.matnat.uio.no> Message-ID: 2011/3/21 Dag Sverre Seljebotn : > On 03/21/2011 11:45 AM, Vitja Makarov wrote: >> >> Now error/warning messages are stored in global variables at >> Cython.Compiler.Errors >> >> I think it's much better to move error handling into some object, >> Main.Context for example. >> >> Some benefits: >> ?- reduce use of global variables >> ?- allow more then one cython compiler instance at the time >> ?- make it much easy to implement -Werror >> ?- cython directives can affect message system (fast_fail, werror) >> ?- messages could be easily sorted > > +1. I assume the reason this is not done is simply because it would be a lot > of work and the payback is less than spending time on other stuff. > > By attaching the error context to "env" and "code" one can avoid a lot of > signature changes. I think transforms should take the context in their > constructor. > It could be something like proxy object that takes compiler directives into account and proxies messages to context. -- vitja. From stefan_ml at behnel.de Mon Mar 21 12:01:47 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 21 Mar 2011 12:01:47 +0100 Subject: [Cython] Message system refactoring In-Reply-To: References: Message-ID: <4D87301B.5000808@behnel.de> Vitja Makarov, 21.03.2011 11:45: > Now error/warning messages are stored in global variables at > Cython.Compiler.Errors Pyrex legacy. Even the visible warning level is a global variable ("LEVEL"). That's a very bad interface. > I think it's much better to move error handling into some object, > Main.Context for example. > > Some benefits: > - reduce use of global variables > - allow more then one cython compiler instance at the time > - make it much easy to implement -Werror > - cython directives can affect message system (fast_fail, werror) > - messages could be easily sorted +1 Stefan From robertwb at math.washington.edu Mon Mar 21 17:18:57 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 21 Mar 2011 09:18:57 -0700 Subject: [Cython] Message system refactoring In-Reply-To: <4D872D9E.4000905@student.matnat.uio.no> References: <4D872D9E.4000905@student.matnat.uio.no> Message-ID: On Mon, Mar 21, 2011 at 3:51 AM, Dag Sverre Seljebotn wrote: > On 03/21/2011 11:45 AM, Vitja Makarov wrote: >> >> Now error/warning messages are stored in global variables at >> Cython.Compiler.Errors >> >> I think it's much better to move error handling into some object, >> Main.Context for example. >> >> Some benefits: >> ?- reduce use of global variables >> ?- allow more then one cython compiler instance at the time >> ?- make it much easy to implement -Werror >> ?- cython directives can affect message system (fast_fail, werror) >> ?- messages could be easily sorted > > +1. I assume the reason this is not done is simply because it would be a lot > of work and the payback is less than spending time on other stuff. +1. I'm fine with the change, it's just never been as much of a pain point as many other things. > By attaching the error context to "env" and "code" one can avoid a lot of > signature changes. I think transforms should take the context in their > constructor. > > Dag Sverre > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From rafe.kettler at gmail.com Tue Mar 22 02:57:49 2011 From: rafe.kettler at gmail.com (Rafe Kettler) Date: Mon, 21 Mar 2011 21:57:49 -0400 Subject: [Cython] Bug in cython-mode.el under GNU Emacs 23.2.1 Message-ID: I've just tried out cython-mode.el and I was a bit disappointed when it didn't work right off the bat. I added the following to my .emacs: (setq load-path (cons "~/lib/emacs-plugins" load-path)) (require 'cython-mode) this initially gave me the following error when I restarted emacs: Warning (initialization): An error occurred while loading `/home/rafe/.emacs': File error: Cannot open load file, python-mode To ensure normal operation, you should investigate and remove the cause of the error in your initialization file. Start Emacs with the `--debug-init' option to view a complete error backtrace. So, seeing that python-mode wasn't around, I changed line 3 of cython-mode.el from (require 'python-mode) to (require 'python) and everything works now. I got this mode from Github this morning and it is the latest version (you can see it at https://github.com/cython/cython/blob/master/Tools/cython-mode.el). I'm not an emacs lisp expert, so I'm not sure if there's something I'm missing here or if I somehow loaded the package correctly. Anyway, I thought you guys should know that I had a problem with the file but managed to get a fix. If my fix needs to be applied, I'll patch it on Github. This is GNU Emacs 23.2.1 x64 on Fedora 14, btw. Rafe -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitja.makarov at gmail.com Tue Mar 22 06:21:46 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 22 Mar 2011 08:21:46 +0300 Subject: [Cython] Bug in cython-mode.el under GNU Emacs 23.2.1 In-Reply-To: References: Message-ID: 2011/3/22 Rafe Kettler : > I've just tried out cython-mode.el and I was a bit disappointed when it > didn't work right off the bat. > > I added the following to my .emacs: > > (setq load-path (cons "~/lib/emacs-plugins" load-path)) > (require 'cython-mode) > > this initially gave me the following error when I restarted emacs: > > ??? Warning (initialization): An error occurred while loading > `/home/rafe/.emacs': > > ??? File error: Cannot open load file, python-mode > > ??? To ensure normal operation, you should investigate and remove the > ??? cause of the error in your initialization file.? Start Emacs with > ??? the `--debug-init' option to view a complete error backtrace. > > So, seeing that python-mode wasn't around, I changed line 3 of > cython-mode.el from (require 'python-mode) to (require 'python) and > everything works now. > > I got this mode from Github this morning and it is the latest version (you > can see it at > https://github.com/cython/cython/blob/master/Tools/cython-mode.el). > > I'm not an emacs lisp expert, so I'm not sure if there's something I'm > missing here or if I somehow loaded the package correctly. Anyway, I thought > you guys should know that I had a problem with the file but managed to get a > fix. If my fix needs to be applied, I'll patch it on Github. > > This is GNU Emacs 23.2.1 x64 on Fedora 14, btw. > > Rafe > Emacs23 have native support for python, you can install python-mode package: $ sudo apt-get install python-mode Instead of requiring python it's better to try python-mode first then python: (when (not (require 'python-mode nil t)) (require 'python)) -- vitja. From stefan_ml at behnel.de Tue Mar 22 07:10:09 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Mar 2011 07:10:09 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> Message-ID: <4D883D41.1050106@behnel.de> Hi, there seems to be quite some interest in a project to get parts of CPython and specifically its stdlib rewritten in Cython. I've copied the latest python-dev mail below. The relevant part of the thread is here: http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 In short, we have strong supporters, but Guido has understandable doubts against a new (and quite large) dependency and potential semantic deviations. But there seem to be cases where slight changes would be acceptable that Cython compiled modules might introduce, such as emitting different exception messages, changing Python classes into extension classes, or even preventing monkey patching in modules that are backed by C modules anyway. It would be helpful to get support from the side of external distributors that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If they agreed to test the Cython generated stdlib modules in their distributions, we could get user feedback that would allow python-dev to take a well founded decision. Do we have any volunteers for trying this out? Both on the side of distributors and implementors? At the current state of affairs, the implementation could still be financed by a Python backed GSoC project, although it would be cool if more users could just step up and simply try to compile and optimise stdlib modules (preferably without major changes to the code). It's certainly a great way to show off your Cython skills :). I gave it a try with difflib and it turned out to be quite easy. http://blog.behnel.de/index.php?p=155 Reimplementing existing C modules in Cython might, however, be more interesting for python-dev, but also be a larger undertaking. So a GSoC might be worth running on that. Note that the latest Cython release does not have generator support yet, and Vitja's branch on github is not very stable. We will try to get it up to speed and merged during the workshop next week, at which point it will make more sense to get this project started than right now. Stefan Guido van Rossum, 22.03.2011 00:04: > On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. L?wis" wrote: >> Am 21.03.2011 11:58, schrieb Stefan Behnel: >>> Guido van Rossum, 21.03.2011 03:46: >>>> Thanks for the clarifications. I now have a much better understanding >>>> of what Cython is. But I'm not sold. For one, your attitude about >>>> strict language compatibility worries me when it comes to the stdlib. >>> >>> Not sure what you mean exactly. Given our large user base, we do worry a >>> lot about things like backwards compatibility, for example. >>> >>> If you are referring to compatibility with Python, I don't think anyone >>> in the project really targets Cython as a a drop-in replacement for a >>> Python runtime. We aim to compile Python code, yes, and there's a >>> hand-wavy idea in the back of our head that we may want a plain Python >>> compatibility mode at some point that will disable several important >>> optimisations. >> >> I think that's the attitude Guido worries about: if you don't have the >> desire to provide 100% Python compatibility under all circumstances >> (i.e. including if someone passes parameters of "incorrect" types), >> then there is very little chance that we would replace a Python module >> with a Cython-compiled one. >> >> The only exception would be cases where the Python semantics is murky >> (e.g. where Jython or so actually behaves differently for the same >> Python code, and still claims language conformance). E.g. the exact >> message on a TypeError might change when compiling with Cython, >> but the cases in which you get a TypeError must not change. > > One other significant use case is the situation where we have an > optional replacement module written in C (e.g. heapqmodule.c vs. > heapq.py). There are usually many semantic differences between the C > and pure-python module that we don't care about (e.g. monkeypatching > won't work). > > The size of Cython as a dependency and its development speed are still > problems though. In general for the core I don't think we want the > repo to contain generated code that can only be regenerated using a > 3rd party dependency. (True, we have a few generated files, e.g. > configure; but in that case the generator -- autoconf -- is a > standard installed tool on Linux and is used by most open source > projects.) > > Still, I think it would be great if someone tried something like this > for a specific stdlib module and came back with a story about the > experience, rather than having a theoretical discussion about possible > pros and cons. From stefan_ml at behnel.de Tue Mar 22 07:38:59 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Mar 2011 07:38:59 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D883D41.1050106@behnel.de> References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> Message-ID: <4D884403.5040306@behnel.de> Stefan Behnel, 22.03.2011 07:10: > there seems to be quite some interest in a project to get parts of CPython > and specifically its stdlib rewritten in Cython. > [...] I gave it a try with difflib and it turned out to be quite easy. > > http://blog.behnel.de/index.php?p=155 BTW, given how short that patch is, wouldn't that make a nice little presentation for the Open Cython Day? Show off how to quickly optimise a stdlib module with Cython? Stefan From vitja.makarov at gmail.com Tue Mar 22 07:57:15 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 22 Mar 2011 09:57:15 +0300 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D884403.5040306@behnel.de> References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> <4D884403.5040306@behnel.de> Message-ID: 2011/3/22 Stefan Behnel : > Stefan Behnel, 22.03.2011 07:10: >> >> there seems to be quite some interest in a project to get parts of CPython >> and specifically its stdlib rewritten in Cython. >> [...] I gave it a try with difflib and it turned out to be quite easy. >> >> http://blog.behnel.de/index.php?p=155 > > BTW, given how short that patch is, wouldn't that make a nice little > presentation for the Open Cython Day? Show off how to quickly optimise a > stdlib module with Cython? > That should be nice ;) Greate step forward for cython and python. Btw I think that stdlib module cython optimization should be as simple as writting .pxd file with minimal .py or none modifications As in difflib Also it would be good to have: - performance comparison/monitoring infrastructure - more tests on stdlib -- vitja. From robertwb at math.washington.edu Tue Mar 22 08:14:45 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 22 Mar 2011 00:14:45 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D883D41.1050106@behnel.de> References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> Message-ID: On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: > Hi, > > there seems to be quite some interest in a project to get parts of CPython > and specifically its stdlib rewritten in Cython. I've copied the latest > python-dev mail below. The relevant part of the thread is here: > > http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 Interesting. > In short, we have strong supporters, but Guido has understandable doubts > against a new (and quite large) dependency and potential semantic > deviations. Reading the list, I think others on the list overestimate the semantic differences. Mostly we're talking about things like is vs. equality for floating point numbers and tracebacks (at least for un-annotated code). It's a valid point that Cython is still under such active development. > But there seem to be cases where slight changes would be > acceptable that Cython compiled modules might introduce, such as emitting > different exception messages, changing Python classes into extension > classes, or even preventing monkey patching in modules that are backed by C > modules anyway. > > It would be helpful to get support from the side of external distributors > that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If > they agreed to test the Cython generated stdlib modules in their > distributions, we could get user feedback that would allow python-dev to > take a well founded decision. > > Do we have any volunteers for trying this out? Both on the side of > distributors and implementors? I think Sage might be willing to give it a try. I'll ask tomorrow as part of a talk I'm giving. Note that due to the way Python's import mechanism works, it would be easy (as a first pass) to make a "cythonize this Python install" which would just compile (a subset of) the .py files and drop .so files next to them. This would require no messing with the Python build system or distribution, easy to test and benchmark, and be easy to clean up. > At the current state of affairs, the implementation could still be financed > by a Python backed GSoC project, although it would be cool if more users > could just step up and simply try to compile and optimise stdlib modules > (preferably without major changes to the code). It's certainly a great way > to show off your Cython skills :). I gave it a try with difflib and it > turned out to be quite easy. > > http://blog.behnel.de/index.php?p=155 > > Reimplementing existing C modules in Cython might, however, be more > interesting for python-dev, but also be a larger undertaking. So a GSoC > might be worth running on that. I think that's a great idea. Would you be willing to mentor such a project. > Note that the latest Cython release does not have generator support yet, and > Vitja's branch on github is not very stable. We will try to get it up to > speed and merged during the workshop next week, at which point it will make > more sense to get this project started than right now. > > Stefan > > > Guido van Rossum, 22.03.2011 00:04: >> >> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. L?wis" wrote: >>> >>> Am 21.03.2011 11:58, schrieb Stefan Behnel: >>>> >>>> Guido van Rossum, 21.03.2011 03:46: >>>>> >>>>> Thanks for the clarifications. I now have a much better understanding >>>>> of what Cython is. But I'm not sold. For one, your attitude about >>>>> strict language compatibility worries me when it comes to the stdlib. >>>> >>>> Not sure what you mean exactly. Given our large user base, we do worry a >>>> lot about things like backwards compatibility, for example. >>>> >>>> If you are referring to compatibility with Python, I don't think anyone >>>> in the project really targets Cython as a a drop-in replacement for a >>>> Python runtime. We aim to compile Python code, yes, and there's a >>>> hand-wavy idea in the back of our head that we may want a plain Python >>>> compatibility mode at some point that will disable several important >>>> optimisations. >>> >>> I think that's the attitude Guido worries about: if you don't have the >>> desire to provide 100% Python compatibility under all circumstances >>> (i.e. including if someone passes parameters of "incorrect" types), >>> then there is very little chance that we would replace a Python module >>> with a Cython-compiled one. >>> >>> The only exception would be cases where the Python semantics is murky >>> (e.g. where Jython or so actually behaves differently for the same >>> ?Python code, and still claims language conformance). E.g. the exact >>> message on a TypeError might change when compiling with Cython, >>> but the cases in which you get a TypeError must not change. >> >> One other significant use case is the situation where we have an >> optional replacement module written in C (e.g. heapqmodule.c vs. >> heapq.py). There are usually many semantic differences between the C >> and pure-python module that we don't care about (e.g. monkeypatching >> won't work). >> >> The size of Cython as a dependency and its development speed are still >> problems though. In general for the core I don't think we want the >> repo to contain generated code that can only be regenerated using a >> 3rd party dependency. (True, we have a few generated files, e.g. >> configure; but in that case the generator -- autoconf -- ?is a >> standard installed tool on Linux and is used by most open source >> projects.) >> >> Still, I think it would be great if someone tried something like this >> for a specific stdlib module and came back with a story about the >> experience, rather than having a theoretical discussion about possible >> pros and cons. > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Tue Mar 22 08:59:04 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Mar 2011 08:59:04 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> Message-ID: <4D8856C8.8050803@behnel.de> Robert Bradshaw, 22.03.2011 08:14: > On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >> there seems to be quite some interest in a project to get parts of CPython >> and specifically its stdlib rewritten in Cython. [...] >> In short, we have strong supporters, but Guido has understandable doubts >> against a new (and quite large) dependency and potential semantic >> deviations. > > Reading the list, I think others on the list overestimate the semantic > differences. Mostly we're talking about things like is vs. equality > for floating point numbers and tracebacks (at least for un-annotated > code). I think so too, that's what I tried to make clearer with my last reply. I think Cython is actually pretty close to Python semantics overall, and almost all deviations are explicitly triggered by type annotations in the code. > It's a valid point that Cython is still under such active development. Absolutely. Eventually, we'd have to settle on a specific version for the compiler used in the stdlib, and support that at least as long as the CPython version that uses it. >> It would be helpful to get support from the side of external distributors >> that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If >> they agreed to test the Cython generated stdlib modules in their >> distributions, we could get user feedback that would allow python-dev to >> take a well founded decision. >> >> Do we have any volunteers for trying this out? Both on the side of >> distributors and implementors? > > I think Sage might be willing to give it a try. I'll ask tomorrow as > part of a talk I'm giving. Cool. > Note that due to the way Python's import > mechanism works, it would be easy (as a first pass) to make a > "cythonize this Python install" which would just compile (a subset of) > the .py files and drop .so files next to them. This would require no > messing with the Python build system or distribution, easy to test and > benchmark, and be easy to clean up. Sure, I implemented that in pyximport ages ago, when Cython was really far from being able to compile much in the stdlib. We should totally give it a try once Vitja's branch is in shape. >> At the current state of affairs, the implementation could still be financed >> by a Python backed GSoC project, although it would be cool if more users >> could just step up and simply try to compile and optimise stdlib modules >> (preferably without major changes to the code). It's certainly a great way >> to show off your Cython skills :). I gave it a try with difflib and it >> turned out to be quite easy. >> >> http://blog.behnel.de/index.php?p=155 >> >> Reimplementing existing C modules in Cython might, however, be more >> interesting for python-dev, but also be a larger undertaking. So a GSoC >> might be worth running on that. > > I think that's a great idea. Would you be willing to mentor such a project. As usual, I'm not sure I'll have the time, but if no-one else steps up, I'd consider it. Stefan From jason-sage at creativetrax.com Tue Mar 22 15:33:37 2011 From: jason-sage at creativetrax.com (Jason Grout) Date: Tue, 22 Mar 2011 09:33:37 -0500 Subject: [Cython] PyCharm In-Reply-To: References: Message-ID: <4D88B341.1090409@creativetrax.com> On 3/16/11 5:33 AM, mark florisson wrote: > The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the > Cython project a free Open Source license, which means that anyone > developing Cython may freely use PyCharm to develop Cython. They > prefer to license to remain unpublic, so if you develop Cython and > want a free PyCharm license, send me an email and I will send you the > license key. It is valid until 13 March 2012. Is it the offer that is valid for a year, or is it the license that is valid for only a year? Thanks, Jason From anders at embl.de Tue Mar 22 15:34:02 2011 From: anders at embl.de (Simon Anders) Date: Tue, 22 Mar 2011 15:34:02 +0100 Subject: [Cython] cython crash: default parameters in methods of template cppclass Message-ID: <4D88B35A.1000408@embl.de> Hi, I found a little bug in Cython 0.14.1. The following causes the Cython compiler to throw an exception: ---8<--- cdef extern from "foo.h": cdef cppclass foo[ T ]: bar( int b = 0 ) cdef foo[ int ] a a.bar( 1 ) ---8<--- The exception is "AttributeError: 'CFuncType' object has no attribute 'op_arg_struct'", thrown in line 3044, in generate_result_code. Full stack trace attached. The file compiles without crash if one either - removes the last line, i.e. the call to the class's method, or - removes the template argument, i.e., the "[ T ]" in line 2 and the "[ int ]" in line 5. Hence, the issue only appears when calling methods of _templated_ classes, which have a _default_ value for their argument. Simon +--- | Dr. Simon Anders, Dipl.-Phys. | European Molecular Biology Laboratory (EMBL), Heidelberg | office phone +49-6221-387-8632 | preferred (permanent) e-mail: sanders at fs.tum.de -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: trace URL: From markflorisson88 at gmail.com Tue Mar 22 16:50:23 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 22 Mar 2011 16:50:23 +0100 Subject: [Cython] PyCharm In-Reply-To: <4D88B341.1090409@creativetrax.com> References: <4D88B341.1090409@creativetrax.com> Message-ID: On 22 March 2011 15:33, Jason Grout wrote: > On 3/16/11 5:33 AM, mark florisson wrote: >> >> The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the >> Cython project a free Open Source license, which means that anyone >> developing Cython may freely use PyCharm to develop Cython. They >> prefer to license to remain unpublic, so if you develop Cython and >> want a free PyCharm license, send me an email and I will send you the >> license key. It is valid until 13 March 2012. > > > Is it the offer that is valid for a year, or is it the license that is valid > for only a year? The license, after that we have to renew it. > Thanks, > > Jason > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From drsalists at gmail.com Wed Mar 23 00:09:44 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 22 Mar 2011 16:09:44 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D883D41.1050106@behnel.de> References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> Message-ID: I think it's a good idea, but I think it'd be better to use pure mode to get code that runs either way, or some sort of preprocessor (I've used m4 with good luck for this, though it doesn't syntax highlight nicely) to automatically derive pure python and cython from the same source file. For me at least, the branch of Cython that supports generators has worked flawlessly - except for one buglet that prevented use on 2.5.x. It was quickly fixed when reported though. Cython probably should get out of the tiny-version-number phase first though. I often feel that opensource projects use tiny version numbers for years as a sort of cop out. well after people have begun relying on the code for production use. On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: > Hi, > > there seems to be quite some interest in a project to get parts of CPython > and specifically its stdlib rewritten in Cython. I've copied the latest > python-dev mail below. The relevant part of the thread is here: > > http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 > > In short, we have strong supporters, but Guido has understandable doubts > against a new (and quite large) dependency and potential semantic > deviations. But there seem to be cases where slight changes would be > acceptable that Cython compiled modules might introduce, such as emitting > different exception messages, changing Python classes into extension > classes, or even preventing monkey patching in modules that are backed by C > modules anyway. > > It would be helpful to get support from the side of external distributors > that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If > they agreed to test the Cython generated stdlib modules in their > distributions, we could get user feedback that would allow python-dev to > take a well founded decision. > > Do we have any volunteers for trying this out? Both on the side of > distributors and implementors? > > At the current state of affairs, the implementation could still be financed > by a Python backed GSoC project, although it would be cool if more users > could just step up and simply try to compile and optimise stdlib modules > (preferably without major changes to the code). It's certainly a great way > to show off your Cython skills :). I gave it a try with difflib and it > turned out to be quite easy. > > http://blog.behnel.de/index.php?p=155 > > Reimplementing existing C modules in Cython might, however, be more > interesting for python-dev, but also be a larger undertaking. So a GSoC > might be worth running on that. > > Note that the latest Cython release does not have generator support yet, > and Vitja's branch on github is not very stable. We will try to get it up to > speed and merged during the workshop next week, at which point it will make > more sense to get this project started than right now. > > Stefan > > > Guido van Rossum, 22.03.2011 00:04: > >> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. L?wis" wrote: >> >>> Am 21.03.2011 11:58, schrieb Stefan Behnel: >>> >>>> Guido van Rossum, 21.03.2011 03:46: >>>> >>>>> Thanks for the clarifications. I now have a much better understanding >>>>> of what Cython is. But I'm not sold. For one, your attitude about >>>>> strict language compatibility worries me when it comes to the stdlib. >>>>> >>>> >>>> Not sure what you mean exactly. Given our large user base, we do worry a >>>> lot about things like backwards compatibility, for example. >>>> >>>> If you are referring to compatibility with Python, I don't think anyone >>>> in the project really targets Cython as a a drop-in replacement for a >>>> Python runtime. We aim to compile Python code, yes, and there's a >>>> hand-wavy idea in the back of our head that we may want a plain Python >>>> compatibility mode at some point that will disable several important >>>> optimisations. >>>> >>> >>> I think that's the attitude Guido worries about: if you don't have the >>> desire to provide 100% Python compatibility under all circumstances >>> (i.e. including if someone passes parameters of "incorrect" types), >>> then there is very little chance that we would replace a Python module >>> with a Cython-compiled one. >>> >>> The only exception would be cases where the Python semantics is murky >>> (e.g. where Jython or so actually behaves differently for the same >>> Python code, and still claims language conformance). E.g. the exact >>> message on a TypeError might change when compiling with Cython, >>> but the cases in which you get a TypeError must not change. >>> >> >> One other significant use case is the situation where we have an >> optional replacement module written in C (e.g. heapqmodule.c vs. >> heapq.py). There are usually many semantic differences between the C >> and pure-python module that we don't care about (e.g. monkeypatching >> won't work). >> >> The size of Cython as a dependency and its development speed are still >> problems though. In general for the core I don't think we want the >> repo to contain generated code that can only be regenerated using a >> 3rd party dependency. (True, we have a few generated files, e.g. >> configure; but in that case the generator -- autoconf -- is a >> standard installed tool on Linux and is used by most open source >> projects.) >> >> Still, I think it would be great if someone tried something like this >> for a specific stdlib module and came back with a story about the >> experience, rather than having a theoretical discussion about possible >> pros and cons. >> > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at math.washington.edu Wed Mar 23 00:54:30 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 22 Mar 2011 16:54:30 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> Message-ID: On Tue, Mar 22, 2011 at 4:09 PM, Dan Stromberg wrote: > > I think it's a good idea, but I think it'd be better to use pure mode to get > code that runs either way, or some sort of preprocessor (I've used m4 with > good luck for this, though it doesn't syntax highlight nicely) to > automatically derive pure python and cython from the same source file. It doesn't hurt to explore the potential before coming up with the actual solution. Ideally, the .py files would not have to be modified at all. > For me at least, the branch of Cython that supports generators has worked > flawlessly - except for one buglet that prevented use on 2.5.x.? It was > quickly fixed when reported though. > > Cython probably should get out of the tiny-version-number phase first > though.? I often feel that opensource projects use tiny version numbers for > years as a sort of cop out. well after people have begun relying on the code > for production use. We have a clear 1.0 goal, being able to compile the full Python language. We're not there yet, but very close. It may make sense at that point to also clean up any cruft we don't want to continue supporting forever. I agree, until that point, there's no way we would be a Python development dependency. - Robert > On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >> >> Hi, >> >> there seems to be quite some interest in a project to get parts of CPython >> and specifically its stdlib rewritten in Cython. I've copied the latest >> python-dev mail below. The relevant part of the thread is here: >> >> http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 >> >> In short, we have strong supporters, but Guido has understandable doubts >> against a new (and quite large) dependency and potential semantic >> deviations. But there seem to be cases where slight changes would be >> acceptable that Cython compiled modules might introduce, such as emitting >> different exception messages, changing Python classes into extension >> classes, or even preventing monkey patching in modules that are backed by C >> modules anyway. >> >> It would be helpful to get support from the side of external distributors >> that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If >> they agreed to test the Cython generated stdlib modules in their >> distributions, we could get user feedback that would allow python-dev to >> take a well founded decision. >> >> Do we have any volunteers for trying this out? Both on the side of >> distributors and implementors? >> >> At the current state of affairs, the implementation could still be >> financed by a Python backed GSoC project, although it would be cool if more >> users could just step up and simply try to compile and optimise stdlib >> modules (preferably without major changes to the code). It's certainly a >> great way to show off your Cython skills :). I gave it a try with difflib >> and it turned out to be quite easy. >> >> http://blog.behnel.de/index.php?p=155 >> >> Reimplementing existing C modules in Cython might, however, be more >> interesting for python-dev, but also be a larger undertaking. So a GSoC >> might be worth running on that. >> >> Note that the latest Cython release does not have generator support yet, >> and Vitja's branch on github is not very stable. We will try to get it up to >> speed and merged during the workshop next week, at which point it will make >> more sense to get this project started than right now. >> >> Stefan >> >> >> Guido van Rossum, 22.03.2011 00:04: >>> >>> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. L?wis" wrote: >>>> >>>> Am 21.03.2011 11:58, schrieb Stefan Behnel: >>>>> >>>>> Guido van Rossum, 21.03.2011 03:46: >>>>>> >>>>>> Thanks for the clarifications. I now have a much better understanding >>>>>> of what Cython is. But I'm not sold. For one, your attitude about >>>>>> strict language compatibility worries me when it comes to the stdlib. >>>>> >>>>> Not sure what you mean exactly. Given our large user base, we do worry >>>>> a >>>>> lot about things like backwards compatibility, for example. >>>>> >>>>> If you are referring to compatibility with Python, I don't think anyone >>>>> in the project really targets Cython as a a drop-in replacement for a >>>>> Python runtime. We aim to compile Python code, yes, and there's a >>>>> hand-wavy idea in the back of our head that we may want a plain Python >>>>> compatibility mode at some point that will disable several important >>>>> optimisations. >>>> >>>> I think that's the attitude Guido worries about: if you don't have the >>>> desire to provide 100% Python compatibility under all circumstances >>>> (i.e. including if someone passes parameters of "incorrect" types), >>>> then there is very little chance that we would replace a Python module >>>> with a Cython-compiled one. >>>> >>>> The only exception would be cases where the Python semantics is murky >>>> (e.g. where Jython or so actually behaves differently for the same >>>> ?Python code, and still claims language conformance). E.g. the exact >>>> message on a TypeError might change when compiling with Cython, >>>> but the cases in which you get a TypeError must not change. >>> >>> One other significant use case is the situation where we have an >>> optional replacement module written in C (e.g. heapqmodule.c vs. >>> heapq.py). There are usually many semantic differences between the C >>> and pure-python module that we don't care about (e.g. monkeypatching >>> won't work). >>> >>> The size of Cython as a dependency and its development speed are still >>> problems though. In general for the core I don't think we want the >>> repo to contain generated code that can only be regenerated using a >>> 3rd party dependency. (True, we have a few generated files, e.g. >>> configure; but in that case the generator -- autoconf -- ?is a >>> standard installed tool on Linux and is used by most open source >>> projects.) >>> >>> Still, I think it would be great if someone tried something like this >>> for a specific stdlib module and came back with a story about the >>> experience, rather than having a theoretical discussion about possible >>> pros and cons. >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > From craigcitro at gmail.com Wed Mar 23 08:11:09 2011 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 23 Mar 2011 00:11:09 -0700 (PDT) Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: Message-ID: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> > We have a clear 1.0 goal, being able to compile the full Python > language. We're not there yet, but very close. It may make sense at > that point to also clean up any cruft we don't want to continue > supporting forever. I agree, until that point, there's no way we would > be a Python development dependency. Are we aiming for 100% compatibility, or 99.9%? For instance, I seem to recall a few dark corners we aren't planning on covering -- maybe some details of traceback construction? (I want to say multiple inheritance, too, but I think that's only an issue for cdef classes, right?) I think it would be good to have this written down -- in particular, it seems like there's some momentum right now for clearly delineating "Python language semantics" vs. "CPython implementation detail" in the python-devel community, so it might be a particularly good time to raise these questions. -cc -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Mar 23 08:31:07 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Mar 2011 08:31:07 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> Message-ID: <4D89A1BB.1080404@behnel.de> Robert Bradshaw, 23.03.2011 00:54: > On Tue, Mar 22, 2011 at 4:09 PM, Dan Stromberg wrote: >> >> I think it's a good idea, but I think it'd be better to use pure mode to get >> code that runs either way, or some sort of preprocessor (I've used m4 with >> good luck for this, though it doesn't syntax highlight nicely) to >> automatically derive pure python and cython from the same source file. > > It doesn't hurt to explore the potential before coming up with the > actual solution. Ideally, the .py files would not have to be modified > at all. Or only slightly, in an acceptable way (whatever that means in a given context). For difflib, for example, I could clean up a couple of things that I'd also frown upon in Python code, like taking off a bound method for __contains__, instead of using a straight and obvious 'in' test. >> For me at least, the branch of Cython that supports generators has worked >> flawlessly It certainly has bugs. For example, I get C compiler warnings when compiling the optimised difflib. And it has disabled one of my favourite features, inlined generator expressions. We'll try to fix it up during the workshop next week. Stefan From stefan_ml at behnel.de Wed Mar 23 08:45:01 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Mar 2011 08:45:01 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> Message-ID: <4D89A4FD.1060408@behnel.de> Craig Citro, 23.03.2011 08:11: >> We have a clear 1.0 goal, being able to compile the full Python >> language. We're not there yet, but very close. It may make sense at >> that point to also clean up any cruft we don't want to continue >> supporting forever. I agree, until that point, there's no way we would >> be a Python development dependency. > > Are we aiming for 100% compatibility, or 99.9%? For instance, I seem to > recall a few dark corners we aren't planning on covering I think it's more like 98%. CPython, PyPy and friends consider the more obscure features like frame access a core Python language feature, but I don't see an interest in making that work in Cython. If someone wants to write the code, fine, as long as it's switched off outside of a future "strict compatibility mode". I've never seen this used in any code that I would have wanted to compile with Cython, and even in Python code it's only really used for more or less "clever hacks". > maybe some details of traceback construction? What I would like to see here is simply a fallback of the frames that we create for tracebacks to look for a .py file (or even .pyx/.pxi file) next to the compiled .so file, so that they can print the source line for the executed line in the .so. Apart from that, it boils down to the issue of frame construction. See above. Keep in mind that CPython's own C code currently gives you no frames at all, so Cython modules already have a serious advantage over what's there today. > (I want to say multiple inheritance, too, > but I think that's only an issue for cdef classes, right?) Yes. That's how things work in CPython at the C level, and I'm fine with that. > I think it would > be good to have this written down -- in particular, it seems like there's > some momentum right now for clearly delineating "Python language semantics" > vs. "CPython implementation detail" in the python-devel community, so it > might be a particularly good time to raise these questions. +1, a "where do we stand?" topic is planned for the workshop anyway. Stefan From vitja.makarov at gmail.com Wed Mar 23 08:52:08 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 23 Mar 2011 10:52:08 +0300 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D89A4FD.1060408@behnel.de> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> Message-ID: 2011/3/23 Stefan Behnel : > Craig Citro, 23.03.2011 08:11: >>> >>> We have a clear 1.0 goal, being able to compile the full Python >>> language. We're not there yet, but very close. It may make sense at >>> that point to also clean up any cruft we don't want to continue >>> supporting forever. I agree, until that point, there's no way we would >>> be a Python development dependency. >> >> Are we aiming for 100% compatibility, or 99.9%? For instance, I seem to >> recall a few dark corners we aren't planning on covering > > I think it's more like 98%. CPython, PyPy and friends consider the more > obscure features like frame access a core Python language feature, but I > don't see an interest in making that work in Cython. If someone wants to > write the code, fine, as long as it's switched off outside of a future > "strict compatibility mode". I've never seen this used in any code that I > would have wanted to compile with Cython, and even in Python code it's only > really used for more or less "clever hacks". > > >> maybe some details of traceback construction? > > What I would like to see here is simply a fallback of the frames that we > create for tracebacks to look for a .py file (or even .pyx/.pxi file) next > to the compiled .so file, so that they can print the source line for the > executed line in the .so. > > Apart from that, it boils down to the issue of frame construction. See > above. Keep in mind that CPython's own C code currently gives you no frames > at all, so Cython modules already have a serious advantage over what's there > today. > > >> (I want to say multiple inheritance, too, >> but I think that's only an issue for cdef classes, right?) > > Yes. That's how things work in CPython at the C level, and I'm fine with > that. > > >> I think it would >> be good to have this written down -- in particular, it seems like there's >> some momentum right now for clearly delineating "Python language >> semantics" >> vs. "CPython implementation detail" in the python-devel community, so it >> might be a particularly good time to raise these questions. > > +1, a "where do we stand?" topic is planned for the workshop anyway. > One more interesting thing is introspection support via inspect module -- vitja. From r.rex at tu-bs.de Wed Mar 23 14:20:52 2011 From: r.rex at tu-bs.de (=?ISO-8859-1?Q?Ren=E9_Rex?=) Date: Wed, 23 Mar 2011 14:20:52 +0100 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: References: Message-ID: Hello The attached code creates a segfault with Cython 0.14.1. A workaround is to use a real function instead of lambda. Using the normal python interpreter the same code works flawlessly. Here are my build commands: cython --embed vectorizeBug.pyx gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O0 -Wall -fPIC -lpython2.6 -I/usr/include/python2.6 -o vectorizeBug vectorizeBug.c -- Ren? Rex TU Braunschweig Bioinformatics & Biochemistry Langer Kamp 19 b 38106 Braunschweig Phone: +49-531-391-8315 -------------- next part -------------- (gdb) run Starting program: /home/user/Desktop/vecBug/vectorizeBug [Thread debugging using libthread_db enabled] [ 2. 2. 2.] Program received signal SIGSEGV, Segmentation fault. 0x00007ffff797b208 in ?? () from /usr/lib/libpython2.6.so.1.0 (gdb) backtrace #0 0x00007ffff797b208 in ?? () from /usr/lib/libpython2.6.so.1.0 #1 0x00007ffff797b590 in PyNumber_Add () from /usr/lib/libpython2.6.so.1.0 #2 0x0000000000401eb5 in __pyx_lambda_funcdef12vectorizeBug_4main_lambda1 (__pyx_self=0x0, __pyx_args=0x7ffff7f8f050, __pyx_kwds=0x0) at vectorizeBug.c:586 #3 0x00007ffff797abb3 in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #4 0x00007ffff7a202a9 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #5 0x00007ffff7a21468 in PyEval_EvalFrameEx () from /usr/lib/libpython2.6.so.1.0 #6 0x00007ffff7a228b0 in PyEval_EvalCodeEx () from /usr/lib/libpython2.6.so.1.0 #7 0x00007ffff79a8630 in ?? () from /usr/lib/libpython2.6.so.1.0 #8 0x00007ffff797abb3 in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #9 0x00007ffff798c5bf in ?? () from /usr/lib/libpython2.6.so.1.0 #10 0x00007ffff797abb3 in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #11 0x00007ffff79e1670 in ?? () from /usr/lib/libpython2.6.so.1.0 #12 0x00007ffff79d7958 in ?? () from /usr/lib/libpython2.6.so.1.0 #13 0x00007ffff797abb3 in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #14 0x0000000000402535 in __pyx_pf_12vectorizeBug_1main (__pyx_self=0x0, unused=0x0) at vectorizeBug.c:689 #15 0x00007ffff797abb3 in PyObject_Call () from /usr/lib/libpython2.6.so.1.0 #16 0x00000000004034ff in initvectorizeBug () at vectorizeBug.c:952 #17 0x0000000000403bc7 in main (argc=1, argv=0x7fffffffdfd8) at vectorizeBug.c:1149 -------------- next part -------------- A non-text attachment was scrubbed... Name: vectorizeBug.pyx Type: application/octet-stream Size: 451 bytes Desc: not available URL: From vitja.makarov at gmail.com Wed Mar 23 17:25:45 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 23 Mar 2011 19:25:45 +0300 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: References: Message-ID: 2011/3/23 Ren? Rex : > Hello > > The attached code creates a segfault with Cython 0.14.1. A workaround > is to use a real function instead of lambda. Using the normal python > interpreter the same code works flawlessly. > > Here are my build commands: > > cython --embed vectorizeBug.pyx > gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O0 -Wall -fPIC > -lpython2.6 -I/usr/include/python2.6 -o vectorizeBug vectorizeBug.c > > -- > Ren? Rex > TU Braunschweig > Bioinformatics & Biochemistry > Langer Kamp 19 b > 38106 Braunschweig > Phone: +49-531-391-8315 > This is simplified code that segfaults too: def f(): return lambda x=0: x f()() Also I found that this code doesn't even compiles: # Module level lambda with default argument value lambda x=0: x File "../cython.py", line 8, in main(command_line = 1) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line 818, in main result = compile(sources, options) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line 793, in compile return compile_multiple(source, options) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line 765, in compile_multiple result = run_pipeline(source, options) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line 629, in run_pipeline err, enddata = context.run_pipeline(pipeline, source) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line 248, in run_pipeline data = phase(data) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line 571, in parse tree = context.parse(source_desc, scope, pxd = 0, full_module_name = full_module_name) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line 510, in parse tree = Parsing.p_module(s, pxd, full_module_name) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 2816, in p_module body = p_statement_list(s, Ctx(level = level), first_statement = 1) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 1778, in p_statement_list stats.append(p_statement(s, ctx, first_statement = first_statement)) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 1771, in p_statement s, ctx, first_statement = first_statement) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 1632, in p_simple_statement_list stat = p_simple_statement(s, first_statement = first_statement) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 1626, in p_simple_statement node = p_expression_or_assignment(s) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 1049, in p_expression_or_assignment expr_list = [p_testlist_star_expr(s)] File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 1005, in p_testlist_star_expr expr = p_test_or_starred_expr(s) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 191, in p_test_or_starred_expr return p_test(s) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 121, in p_test return p_lambdef(s) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 101, in p_lambdef s, terminator=':', annotated=False) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 2612, in p_varargslist annotated = annotated) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 2261, in p_c_arg_list annotated = annotated)) File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", line 2307, in p_c_arg_decl if 'pxd' in s.level: AttributeError: 'PyrexScanner' object has no attribute 'level' -- vitja. From stefan_ml at behnel.de Wed Mar 23 17:44:42 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Mar 2011 17:44:42 +0100 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: References: Message-ID: <4D8A237A.7000403@behnel.de> Vitja Makarov, 23.03.2011 17:25: > File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", > line 2307, in p_c_arg_decl > if 'pxd' in s.level: > AttributeError: 'PyrexScanner' object has no attribute 'level' Yes it does: """ # Cython/Plex/Scanners.pxd cdef class Scanner: [...] cdef public level """ Try a clean build. Stefa From stefan_ml at behnel.de Wed Mar 23 17:56:26 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 23 Mar 2011 17:56:26 +0100 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: References: Message-ID: <4D8A263A.4050401@behnel.de> Vitja Makarov, 23.03.2011 17:25: > def f(): > return lambda x=0: x > f()() Gives me this bogus code at function entry: Py_ssize_t kw_args = PyDict_Size(__pyx_kwds); PyObject* values[1] = {0}; values[0] = ((PyObject *)0); // <<<<<<< !!! Looks like the default value is considered an int, whereas x is of type object, and there's a coercion missing between the two. Stefan From vitja.makarov at gmail.com Wed Mar 23 20:11:52 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 23 Mar 2011 22:11:52 +0300 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: <4D8A237A.7000403@behnel.de> References: <4D8A237A.7000403@behnel.de> Message-ID: 2011/3/23 Stefan Behnel : > Vitja Makarov, 23.03.2011 17:25: >> >> ? File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", >> line 2307, in p_c_arg_decl >> ? ? if 'pxd' in s.level: >> AttributeError: 'PyrexScanner' object has no attribute 'level' > > Yes it does: > > """ > # Cython/Plex/Scanners.pxd > cdef class Scanner: > ? ?[...] > ? ?cdef public level > """ > > Try a clean build. > Don't actually understand what's going wrong here? I'm running plain python cython, probably I'm wrong. lambda x=0:0 # give same error, but lambda x:0 work fine About bug: it seems that analyse_expressions() is not called for LambdaNode. Hope that will help. -- vitja. From dalcinl at gmail.com Wed Mar 23 23:48:41 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 23 Mar 2011 19:48:41 -0300 Subject: [Cython] Bug in cython-mode.el under GNU Emacs 23.2.1 In-Reply-To: References: Message-ID: On 22 March 2011 02:21, Vitja Makarov wrote: > 2011/3/22 Rafe Kettler : >> I've just tried out cython-mode.el and I was a bit disappointed when it >> didn't work right off the bat. >> >> I added the following to my .emacs: >> >> (setq load-path (cons "~/lib/emacs-plugins" load-path)) >> (require 'cython-mode) >> >> this initially gave me the following error when I restarted emacs: >> >> ??? Warning (initialization): An error occurred while loading >> `/home/rafe/.emacs': >> >> ??? File error: Cannot open load file, python-mode >> >> ??? To ensure normal operation, you should investigate and remove the >> ??? cause of the error in your initialization file.? Start Emacs with >> ??? the `--debug-init' option to view a complete error backtrace. >> >> So, seeing that python-mode wasn't around, I changed line 3 of >> cython-mode.el from (require 'python-mode) to (require 'python) and >> everything works now. >> >> I got this mode from Github this morning and it is the latest version (you >> can see it at >> https://github.com/cython/cython/blob/master/Tools/cython-mode.el). >> >> I'm not an emacs lisp expert, so I'm not sure if there's something I'm >> missing here or if I somehow loaded the package correctly. Anyway, I thought >> you guys should know that I had a problem with the file but managed to get a >> fix. If my fix needs to be applied, I'll patch it on Github. >> >> This is GNU Emacs 23.2.1 x64 on Fedora 14, btw. >> >> Rafe >> > > Emacs23 have native support for python, you can install python-mode package: > > $ sudo apt-get install python-mode > > Instead of requiring python it's better to try python-mode first then python: > > (when (not (require 'python-mode nil t)) > ?(require 'python)) > Should we push this fix? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Thu Mar 24 07:07:59 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 24 Mar 2011 07:07:59 +0100 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: References: <4D8A237A.7000403@behnel.de> Message-ID: <4D8ADFBF.7070709@behnel.de> Vitja Makarov, 23.03.2011 20:11: > 2011/3/23 Stefan Behnel: >> Vitja Makarov, 23.03.2011 17:25: >>> >>> File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", >>> line 2307, in p_c_arg_decl >>> if 'pxd' in s.level: >>> AttributeError: 'PyrexScanner' object has no attribute 'level' >> >> Yes it does: >> >> """ >> # Cython/Plex/Scanners.pxd >> cdef class Scanner: >> [...] >> cdef public level >> """ >> >> Try a clean build. > > Don't actually understand what's going wrong here? > I'm running plain python cython, probably I'm wrong. These things usually happen to me when I use e.g. a compiled Scanners.so but a plain Parsing.py. Stefan From stefan_ml at behnel.de Thu Mar 24 18:42:55 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 24 Mar 2011 18:42:55 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8856C8.8050803@behnel.de> References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> <4D8856C8.8050803@behnel.de> Message-ID: <4D8B829F.5080402@behnel.de> Stefan Behnel, 22.03.2011 08:59: > Robert Bradshaw, 22.03.2011 08:14: >> On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >>> Reimplementing existing C modules in Cython might, however, be more >>> interesting for python-dev, but also be a larger undertaking. So a GSoC >>> might be worth running on that. >> >> I think that's a great idea. Would you be willing to mentor such a project. > > As usual, I'm not sure I'll have the time, but if no-one else steps up, I'd > consider it. Should we sign up with the PSF GSoC umbrella to officially propose this project? https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUUVGWmc6MQ Do we have any other topics for the GSoC this year? We'd be just in time if we discuss this at the workshop, although somewhat late already. The student application deadline is from March 28 to April 8. http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline Stefan From vitja.makarov at gmail.com Thu Mar 24 19:15:19 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 24 Mar 2011 21:15:19 +0300 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: <4D8ADFBF.7070709@behnel.de> References: <4D8A237A.7000403@behnel.de> <4D8ADFBF.7070709@behnel.de> Message-ID: 2011/3/24 Stefan Behnel : > Vitja Makarov, 23.03.2011 20:11: >> >> 2011/3/23 Stefan Behnel: >>> >>> Vitja Makarov, 23.03.2011 17:25: >>>> >>>> ? File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", >>>> line 2307, in p_c_arg_decl >>>> ? ? if 'pxd' in s.level: >>>> AttributeError: 'PyrexScanner' object has no attribute 'level' >>> >>> Yes it does: >>> >>> """ >>> # Cython/Plex/Scanners.pxd >>> cdef class Scanner: >>> ? ?[...] >>> ? ?cdef public level >>> """ >>> >>> Try a clean build. >> >> Don't actually understand what's going wrong here? >> I'm running plain python cython, probably I'm wrong. > > These things usually happen to me when I use e.g. a compiled Scanners.so but > a plain Parsing.py. > This is actally cython bug PyrexScanner.level attribute is not initialized at module scope. Simple fix is to set level to 'other' or 'unknown' -- vitja. From stefan_ml at behnel.de Thu Mar 24 19:29:28 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 24 Mar 2011 19:29:28 +0100 Subject: [Cython] Bug report: lambda and numpy.vectorize segfaults In-Reply-To: References: <4D8A237A.7000403@behnel.de> <4D8ADFBF.7070709@behnel.de> Message-ID: <4D8B8D88.8030206@behnel.de> Vitja Makarov, 24.03.2011 19:15: > 2011/3/24 Stefan Behnel: >> Vitja Makarov, 23.03.2011 20:11: >>> >>> 2011/3/23 Stefan Behnel: >>>> >>>> Vitja Makarov, 23.03.2011 17:25: >>>>> >>>>> File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py", >>>>> line 2307, in p_c_arg_decl >>>>> if 'pxd' in s.level: >>>>> AttributeError: 'PyrexScanner' object has no attribute 'level' >>>> >>>> Yes it does: >>>> >>>> """ >>>> # Cython/Plex/Scanners.pxd >>>> cdef class Scanner: >>>> [...] >>>> cdef public level >>>> """ >>>> >>>> Try a clean build. >>> >>> Don't actually understand what's going wrong here? >>> I'm running plain python cython, probably I'm wrong. >> >> These things usually happen to me when I use e.g. a compiled Scanners.so but >> a plain Parsing.py. >> > > This is actally cython bug PyrexScanner.level attribute is not > initialized at module scope. Ah, right, I actually appended the test code to the existing lambda tests, that's why the problem didn't show. > Simple fix is to set level to 'other' or 'unknown' Rather 'module' or 'module_pxd', depending on the parsing context. Stefan From robertwb at math.washington.edu Thu Mar 24 20:18:06 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 24 Mar 2011 12:18:06 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8B829F.5080402@behnel.de> References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> <4D8856C8.8050803@behnel.de> <4D8B829F.5080402@behnel.de> Message-ID: On Thu, Mar 24, 2011 at 10:42 AM, Stefan Behnel wrote: > Stefan Behnel, 22.03.2011 08:59: >> >> Robert Bradshaw, 22.03.2011 08:14: >>> >>> On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >>>> >>>> Reimplementing existing C modules in Cython might, however, be more >>>> interesting for python-dev, but also be a larger undertaking. So a GSoC >>>> might be worth running on that. >>> >>> I think that's a great idea. Would you be willing to mentor such a >>> project. >> >> As usual, I'm not sure I'll have the time, but if no-one else steps up, >> I'd >> consider it. > > Should we sign up with the PSF GSoC umbrella to officially propose this > project? > > https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUUVGWmc6MQ > > Do we have any other topics for the GSoC this year? We'd be just in time if > we discuss this at the workshop, although somewhat late already. The student > application deadline is from March 28 to April 8. > > http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline There's always our enhancements page, but some of those are fairly advanced. I think this'd be a really good GSoC project, very modular, and doesn't require getting to know the internals of the compiler to start making a serious contribution (at least not right away, I'm sure bugs and optimizations will be submitted). I'm willing to (co?)-mentor. Stefan? Craig? With the three of us we could easily mentor at least one student, and perhaps two (if another really solid proposal/student comes up). Anyone else willing to mentor? I haven't pushed on GSoC much this year yet because no one's stepped up to mentor, but there's still ample time on our side. - Robert From robertwb at math.washington.edu Thu Mar 24 20:38:23 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 24 Mar 2011 12:38:23 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D89A4FD.1060408@behnel.de> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> Message-ID: On Wed, Mar 23, 2011 at 12:45 AM, Stefan Behnel wrote: > Craig Citro, 23.03.2011 08:11: >>> >>> We have a clear 1.0 goal, being able to compile the full Python >>> language. We're not there yet, but very close. It may make sense at >>> that point to also clean up any cruft we don't want to continue >>> supporting forever. I agree, until that point, there's no way we would >>> be a Python development dependency. >> >> Are we aiming for 100% compatibility, or 99.9%? For instance, I seem to >> recall a few dark corners we aren't planning on covering I started a list at http://wiki.cython.org/Unsupported . I'd say we can be as compatible as Jython/IronPython is, and more than CPython is between minor versions. I would be happy with a short, well-justified list of differences. This will be clearer once the community (which we're a part of) defines what Python vs. implementation details means. > I think it's more like 98%. CPython, PyPy and friends consider the more > obscure features like frame access a core Python language feature, but I > don't see an interest in making that work in Cython. If someone wants to > write the code, fine, as long as it's switched off outside of a future > "strict compatibility mode". I've never seen this used in any code that I > would have wanted to compile with Cython, and even in Python code it's only > really used for more or less "clever hacks". > > >> maybe some details of traceback construction? > > What I would like to see here is simply a fallback of the frames that we > create for tracebacks to look for a .py file (or even .pyx/.pxi file) next > to the compiled .so file, so that they can print the source line for the > executed line in the .so. > > Apart from that, it boils down to the issue of frame construction. See > above. Keep in mind that CPython's own C code currently gives you no frames > at all, so Cython modules already have a serious advantage over what's there > today. > > >> (I want to say multiple inheritance, too, >> but I think that's only an issue for cdef classes, right?) > > Yes. That's how things work in CPython at the C level, and I'm fine with > that. We have multiple inheritance for normal classes, so that's covered. Any auto-cdeffing of classes would have to be done carefully. >> I think it would >> be good to have this written down -- in particular, it seems like there's >> some momentum right now for clearly delineating "Python language >> semantics" >> vs. "CPython implementation detail" in the python-devel community, so it >> might be a particularly good time to raise these questions. > > +1, a "where do we stand?" topic is planned for the workshop anyway. +1 - Robert From stefan_ml at behnel.de Thu Mar 24 20:58:37 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 24 Mar 2011 20:58:37 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> <4D8856C8.8050803@behnel.de> <4D8B829F.5080402@behnel.de> Message-ID: <4D8BA26D.4080808@behnel.de> Robert Bradshaw, 24.03.2011 20:18: > On Thu, Mar 24, 2011 at 10:42 AM, Stefan Behnel wrote: >> Stefan Behnel, 22.03.2011 08:59: >>> >>> Robert Bradshaw, 22.03.2011 08:14: >>>> >>>> On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >>>>> >>>>> Reimplementing existing C modules in Cython might, however, be more >>>>> interesting for python-dev, but also be a larger undertaking. So a GSoC >>>>> might be worth running on that. >>>> >>>> I think that's a great idea. Would you be willing to mentor such a >>>> project. >>> >>> As usual, I'm not sure I'll have the time, but if no-one else steps up, >>> I'd >>> consider it. >> >> Should we sign up with the PSF GSoC umbrella to officially propose this >> project? >> >> https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUUVGWmc6MQ >> >> Do we have any other topics for the GSoC this year? We'd be just in time if >> we discuss this at the workshop, although somewhat late already. The student >> application deadline is from March 28 to April 8. >> >> http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline > > There's always our enhancements page, but some of those are fairly > advanced. I think this'd be a really good GSoC project, very modular, > and doesn't require getting to know the internals of the compiler to > start making a serious contribution (at least not right away, I'm sure > bugs and optimizations will be submitted). Well, almost certainly bugs will be found, at least, and if we get a fix as well, I'd be the more happy. :) > I'm willing to (co?)-mentor. Stefan? Craig? With the three of us we > could easily mentor at least one student, and perhaps two (if another > really solid proposal/student comes up). +1, splitting this will make it easier for everyone. And there will be support from the lists anyway. > Anyone else willing to > mentor? I haven't pushed on GSoC much this year yet because no one's > stepped up to mentor, but there's still ample time on our side. Ok, want to sign up the Cython project then? What became of the Sage proposal, BTW? Will they give a compiled stdlib a try? Stefan From robertwb at math.washington.edu Thu Mar 24 21:03:45 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 24 Mar 2011 13:03:45 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8BA26D.4080808@behnel.de> References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> <4D8856C8.8050803@behnel.de> <4D8B829F.5080402@behnel.de> <4D8BA26D.4080808@behnel.de> Message-ID: On Thu, Mar 24, 2011 at 12:58 PM, Stefan Behnel wrote: > Robert Bradshaw, 24.03.2011 20:18: >> >> On Thu, Mar 24, 2011 at 10:42 AM, Stefan Behnel wrote: >>> >>> Stefan Behnel, 22.03.2011 08:59: >>>> >>>> Robert Bradshaw, 22.03.2011 08:14: >>>>> >>>>> On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >>>>>> >>>>>> Reimplementing existing C modules in Cython might, however, be more >>>>>> interesting for python-dev, but also be a larger undertaking. So a >>>>>> GSoC >>>>>> might be worth running on that. >>>>> >>>>> I think that's a great idea. Would you be willing to mentor such a >>>>> project. >>>> >>>> As usual, I'm not sure I'll have the time, but if no-one else steps up, >>>> I'd >>>> consider it. >>> >>> Should we sign up with the PSF GSoC umbrella to officially propose this >>> project? >>> >>> >>> https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUUVGWmc6MQ >>> >>> Do we have any other topics for the GSoC this year? We'd be just in time >>> if >>> we discuss this at the workshop, although somewhat late already. The >>> student >>> application deadline is from March 28 to April 8. >>> >>> >>> http://socghop.appspot.com/document/show/gsoc_program/google/gsoc2011/timeline >> >> There's always our enhancements page, but some of those are fairly >> advanced. I think this'd be a really good GSoC project, very modular, >> and doesn't require getting to know the internals of the compiler to >> start making a serious contribution (at least not right away, I'm sure >> bugs and optimizations will be submitted). > > Well, almost certainly bugs will be found, at least, and if we get a fix as > well, I'd be the more happy. :) > > >> I'm willing to (co?)-mentor. Stefan? Craig? With the three of us we >> could easily mentor at least one student, and perhaps two (if another >> really solid proposal/student comes up). > > +1, splitting this will make it easier for everyone. And there will be > support from the lists anyway. > > >> Anyone else willing to >> mentor? I haven't pushed on GSoC much this year yet because no one's >> stepped up to mentor, but there's still ample time on our side. > > Ok, want to sign up the Cython project then? Well, I'm talking about under the Python foundation umbrella. > What became of the Sage proposal, BTW? Will they give a compiled stdlib a > try? I completely forgot to ask (too many other things on my mind). We have a fairly extensive test suite, so I think the thing to do would be to try it out and see if anything breaks (which will be a good confidence builder for both Cython and Sage). - Robert From stefan_ml at behnel.de Thu Mar 24 21:26:16 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 24 Mar 2011 21:26:16 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <20110320124049.70610337@pitrou.net> <4D87D4C2.4030101@v.loewis.de> <4D883D41.1050106@behnel.de> <4D8856C8.8050803@behnel.de> <4D8B829F.5080402@behnel.de> <4D8BA26D.4080808@behnel.de> Message-ID: <4D8BA8E8.4080207@behnel.de> Robert Bradshaw, 24.03.2011 21:03: > On Thu, Mar 24, 2011 at 12:58 PM, Stefan Behnel wrote: >> Robert Bradshaw, 24.03.2011 20:18: >>> Anyone else willing to >>> mentor? I haven't pushed on GSoC much this year yet because no one's >>> stepped up to mentor, but there's still ample time on our side. >> >> Ok, want to sign up the Cython project then? > > Well, I'm talking about under the Python foundation umbrella. That's what I meant. https://spreadsheets.google.com/viewform?formkey=dHh3WFNGYzkyWWE0ZjM1eFFoUUVGWmc6MQ >> What became of the Sage proposal, BTW? Will they give a compiled stdlib a >> try? > > I completely forgot to ask (too many other things on my mind). We have > a fairly extensive test suite, so I think the thing to do would be to > try it out and see if anything breaks (which will be a good confidence > builder for both Cython and Sage). Sure. I think the first step is to show results, then we can see who likes them good enough to take the risk. Stefan From robertwb at math.washington.edu Fri Mar 25 01:18:35 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 24 Mar 2011 17:18:35 -0700 Subject: [Cython] Bug in cython-mode.el under GNU Emacs 23.2.1 In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 3:48 PM, Lisandro Dalcin wrote: > On 22 March 2011 02:21, Vitja Makarov wrote: >> 2011/3/22 Rafe Kettler : >>> I've just tried out cython-mode.el and I was a bit disappointed when it >>> didn't work right off the bat. >>> >>> I added the following to my .emacs: >>> >>> (setq load-path (cons "~/lib/emacs-plugins" load-path)) >>> (require 'cython-mode) >>> >>> this initially gave me the following error when I restarted emacs: >>> >>> ??? Warning (initialization): An error occurred while loading >>> `/home/rafe/.emacs': >>> >>> ??? File error: Cannot open load file, python-mode >>> >>> ??? To ensure normal operation, you should investigate and remove the >>> ??? cause of the error in your initialization file.? Start Emacs with >>> ??? the `--debug-init' option to view a complete error backtrace. >>> >>> So, seeing that python-mode wasn't around, I changed line 3 of >>> cython-mode.el from (require 'python-mode) to (require 'python) and >>> everything works now. >>> >>> I got this mode from Github this morning and it is the latest version (you >>> can see it at >>> https://github.com/cython/cython/blob/master/Tools/cython-mode.el). >>> >>> I'm not an emacs lisp expert, so I'm not sure if there's something I'm >>> missing here or if I somehow loaded the package correctly. Anyway, I thought >>> you guys should know that I had a problem with the file but managed to get a >>> fix. If my fix needs to be applied, I'll patch it on Github. >>> >>> This is GNU Emacs 23.2.1 x64 on Fedora 14, btw. >>> >>> Rafe >>> >> >> Emacs23 have native support for python, you can install python-mode package: >> >> $ sudo apt-get install python-mode >> >> Instead of requiring python it's better to try python-mode first then python: >> >> (when (not (require 'python-mode nil t)) >> ?(require 'python)) >> > > Should we push this fix? Makes sense to me. Rafe, can you push your changes and do a pull request? - Robert From sturla at molden.no Fri Mar 25 14:03:54 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 25 Mar 2011 14:03:54 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> Message-ID: <4D8C92BA.1020602@molden.no> Den 24.03.2011 20:38, skrev Robert Bradshaw: > I started a list at http://wiki.cython.org/Unsupported . I'd say we > can be as compatible as Jython/IronPython is, and more than CPython is > between minor versions. I would be happy with a short, well-justified > list of differences. This will be clearer once the community (which > we're a part of) defines what Python vs. implementation details means. > Looking at Guido's comment, Cython must be able to compile all valid Python if this will have any chance of success. Is the plan to include Cython in the standard library? I don't think a large external dependency like Cython will be accepted unless it's a part of the CPython distribution. Why stop with the standard library? Why not implement the whole CPython interpreter in Cython? Sturla From stefan_ml at behnel.de Fri Mar 25 14:42:39 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 25 Mar 2011 14:42:39 +0100 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8C92BA.1020602@molden.no> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> Message-ID: <4D8C9BCF.8080403@behnel.de> Sturla Molden, 25.03.2011 14:03: > Den 24.03.2011 20:38, skrev Robert Bradshaw: >> I started a list at http://wiki.cython.org/Unsupported . I'd say we >> can be as compatible as Jython/IronPython is, and more than CPython is >> between minor versions. I would be happy with a short, well-justified >> list of differences. This will be clearer once the community (which >> we're a part of) defines what Python vs. implementation details means. > > Looking at Guido's comment, Cython must be able to compile all valid Python > if this will have any chance of success. I think there are two levels of required compatibility, the lower one of which applies to a reimplementation of C modules in Cythen. That's the reason why I see that as the more worthwhile goal for now. > Is the plan to include Cython in the standard library? I don't think a > large external dependency like Cython will be accepted unless it's a part > of the CPython distribution. It was the plan a while ago, but the problem is that Cython's evolution (and stability) isn't very much in line with that of CPython. I'm not as pessimistic as you seem to be, though. From the POV of CPython, Cython is basically just a build tool. We could try to come up with a release series that we consider stable enough to base CPython on it. If we agree to maintain that for the whole lifetime of the corresponding CPython release(s), it wouldn't necessarily have to be part of CPython itself (although it could...) > Why stop with the standard library? Why not implement the whole CPython > interpreter in Cython? No-one said we'd stop there. ;) However, going down that route will quickly turn into a problem of boot-strapping. How would you run Cython without a Python interpreter? Would you need an old interpreter installed in order to run Cython to build a new one? Currently, you only need a C compiler and cmmi tools. I could easily understand anyone who'd reject entering into such a recursive dependency. Stefan From robertwb at math.washington.edu Fri Mar 25 19:03:51 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 25 Mar 2011 11:03:51 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8C9BCF.8080403@behnel.de> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> <4D8C9BCF.8080403@behnel.de> Message-ID: On Fri, Mar 25, 2011 at 6:42 AM, Stefan Behnel wrote: > Sturla Molden, 25.03.2011 14:03: >> >> Den 24.03.2011 20:38, skrev Robert Bradshaw: >>> >>> I started a list at http://wiki.cython.org/Unsupported . I'd say we >>> can be as compatible as Jython/IronPython is, and more than CPython is >>> between minor versions. I would be happy with a short, well-justified >>> list of differences. This will be clearer once the community (which >>> we're a part of) defines what Python vs. implementation details means. >> >> Looking at Guido's comment, Cython must be able to compile all valid >> Python if this will have any chance of success. Good thing that's our goal (pending an actual definition of "all valid Python.") > I think there are two levels of required compatibility, the lower one of > which applies to a reimplementation of C modules in Cythen. That's the > reason why I see that as the more worthwhile goal for now. > > >> Is the plan to include Cython in the standard library? I don't think a >> large external dependency like Cython will be accepted unless it's a part >> of the CPython distribution. > > It was the plan a while ago, but the problem is that Cython's evolution (and > stability) isn't very much in line with that of CPython. I'm not as > pessimistic as you seem to be, though. From the POV of CPython, Cython is > basically just a build tool. We could try to come up with a release series > that we consider stable enough to base CPython on it. If we agree to > maintain that for the whole lifetime of the corresponding CPython > release(s), it wouldn't necessarily have to be part of CPython itself > (although it could...) As a first step, CPython would just ship the generated C, only requiring Cython as a development dependency, not a build dependency. This would also be safer from the stability POV. >> Why stop with the standard library? Why not implement the whole CPython >> interpreter in Cython? > > No-one said we'd stop there. ;) Of course the CPython interpreter is a large, fairly-optimized C codebase already, so the pitfalls of just re-writing it for the sake of re-writing it probably outweigh the gains. > However, going down that route will quickly turn into a problem of > boot-strapping. How would you run Cython without a Python interpreter? Would > you need an old interpreter installed in order to run Cython to build a new > one? Currently, you only need a C compiler and cmmi tools. I could easily > understand anyone who'd reject entering into such a recursive dependency. It would be as easy as bootstrapping gcc... :). - Robert From greg.ewing at canterbury.ac.nz Sat Mar 26 01:57:08 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Mar 2011 13:57:08 +1300 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8C92BA.1020602@molden.no> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> Message-ID: <4D8D39E4.1070702@canterbury.ac.nz> Sturla Molden wrote: > Why stop with the standard library? Why not implement the whole CPython > interpreter in Cython? That would be tricky, because the code that Cython generates depends on large chunks of CPython as an infrastructure. -- Greg From sturla at molden.no Mon Mar 28 00:39:02 2011 From: sturla at molden.no (Sturla Molden) Date: Mon, 28 Mar 2011 00:39:02 +0200 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> <4D8C9BCF.8080403@behnel.de> Message-ID: <4D8FBC86.6060906@molden.no> Den 25.03.2011 19:03, skrev Robert Bradshaw: > >>> Looking at Guido's comment, Cython must be able to compile all valid >>> Python if this will have any chance of success. > Good thing that's our goal (pending an actual definition of "all valid Python.") > In lack of a Python language specification it can be hard to tell implementation details from syntax. It sounded though as if Guido was worried about Cython's compatibility with Python, and maybe the Cython dev team's attitude to Python compatibility. Also don't think Cython's main strength in this context was properly clarified in the debate. It is easy to over-focus on "speed", when it's really a matter of "writing Python C extensions easily" -- i.e. without knowing (a lot) about Python's C API, not having to worry about reference counting, and the possibility of using Python code as prototype. Cython is, without comparison, the easiest way of writing C extensions for Python. FWIW, it's easier to use Cython than ctypes. Using Cython instead of the C API will also avoid many programming errors, because a compiler does fewer mistakes than a human. Those aspects are important to communicate, not just "Cython can be as fast as C++". Sturla From cournape at gmail.com Mon Mar 28 06:46:01 2011 From: cournape at gmail.com (David Cournapeau) Date: Mon, 28 Mar 2011 13:46:01 +0900 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8FBC86.6060906@molden.no> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> <4D8C9BCF.8080403@behnel.de> <4D8FBC86.6060906@molden.no> Message-ID: On Mon, Mar 28, 2011 at 7:39 AM, Sturla Molden wrote: > Cython is, > without comparison, the easiest way of writing C extensions for Python. > FWIW, it's easier to use Cython than ctypes. Using Cython instead of the C > API will also avoid many programming errors, because a compiler does fewer > mistakes than a human. Agreed. I did not jump in in the discussion on python-dev because I am not involved in cython development, but I felt that this point may not have been obvious for people unfamiliar with cython. Being able to have less C in python itself sounds like a better goal to me, and even more useful for alternative implementations than compiling the stdlib (porting cython must be easier than porting C in almost every case). One could imagine using cython for most stuff in Modules (I don't know how much from Modules would be needed for cython itself to solve the bootstrap issue). Cython helps making things fast, but it also removes the need to do raw C wrappers in most cases. cheers, David From robertwb at math.washington.edu Tue Mar 29 02:09:08 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Mar 2011 17:09:08 -0700 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: <4D8FBC86.6060906@molden.no> References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> <4D8C9BCF.8080403@behnel.de> <4D8FBC86.6060906@molden.no> Message-ID: On Sun, Mar 27, 2011 at 3:39 PM, Sturla Molden wrote: > Den 25.03.2011 19:03, skrev Robert Bradshaw: >> >>>> Looking at Guido's comment, Cython must be able to compile all valid >>>> Python if this will have any chance of success. >> >> Good thing that's our goal (pending an actual definition of "all valid >> Python.") >> > > In lack of a Python language specification it can be hard to tell > implementation details from syntax. It sounded though as if Guido was > worried about Cython's compatibility with Python, and maybe the Cython dev > team's attitude to Python compatibility. We are very concerned about Python compatibility. > Also don't think Cython's main strength in this context was properly > clarified in the debate. It is easy to over-focus on "speed", when it's > really a matter of "writing Python C extensions easily" -- i.e. without > knowing (a lot) about Python's C API, not having to worry about reference > counting, and the possibility of using Python code as prototype. Cython is, > without comparison, the easiest way of writing C extensions for Python. > FWIW, it's easier to use Cython than ctypes. Using Cython instead of the C > API will also avoid many programming errors, because a compiler does fewer > mistakes than a human. Those aspects are important to communicate, not just > "Cython can be as fast as C++". That is a good point. On the other hand, I don't see re-implementing working C modules written, though probably valuable from a maintenance point of view, as compelling of a use case. - Robert From robertwb at math.washington.edu Tue Mar 29 02:32:59 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 28 Mar 2011 17:32:59 -0700 Subject: [Cython] cython crash: default parameters in methods of template cppclass In-Reply-To: <4D88B06A.5010802@embl.de> References: <4D88B06A.5010802@embl.de> Message-ID: On Tue, Mar 22, 2011 at 7:21 AM, Simon Anders wrote: > Hi, > > I found a little bug in Cython 0.14.1. > > The following causes the Cython compiler to throw an exception: > > ---8<--- > cdef extern from "foo.h": > ? cdef cppclass foo[ T ]: > ? ? ?bar( int b = 0 ) > > cdef foo[ int ] a > a.bar( 1 ) > ---8<--- > > The exception is "AttributeError: 'CFuncType' object has no attribute > 'op_arg_struct'", thrown in line 3044, in generate_result_code. Full stack > trace attached. > > The file compiles without crash if one either > ?- removes the last line, i.e. the call to the class's method, or > ?- removes the template argument, i.e., the "[ T ]" in line 2 and > ? ? ?the "[ int ]" in line 5. > > Hence, the issue only appears when calling methods of _templated_ classes, > which have a _default_ value for their argument. Default arguments in C++ are completely different from default arguments in Cython, this is our bug for not catching that. You have to declare this as an overloaded method cdef extern from "foo.h": cdef cppclass foo[ T ]: bar() bar(int) Btw, unless it returns an object, you probably want to declare the return type of your methods. Perhaps we should make this a warning for extern declarations? - Robert From stefan_ml at behnel.de Tue Mar 29 08:22:42 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 29 Mar 2011 08:22:42 +0200 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> <4D8C9BCF.8080403@behnel.de> <4D8FBC86.6060906@molden.no> Message-ID: <4D917AB2.5050507@behnel.de> Robert Bradshaw, 29.03.2011 02:09: > I don't see re-implementing > working C modules written, though probably valuable from a maintenance > point of view, as compelling of a use case. It would be rather helpful for CPython, though. Many stdlib modules lack dedicated maintainers, and it's likely easier in the Python world to find a maintainer for code that's "mostly Python", than for code that's C plus CPython's C-API. Stefan From arthurdesribeiro at gmail.com Tue Mar 29 09:11:24 2011 From: arthurdesribeiro at gmail.com (Arthur de Souza Ribeiro) Date: Tue, 29 Mar 2011 04:11:24 -0300 Subject: [Cython] Interest in contributing to the project Message-ID: Hello everybody, My name is Arthur de Souza Ribeiro and I'm a fourth-year student of Computer Science in Federal University of Campina Grande, Brazil. I'm a python programmer and have knowledge of other languages too, like Java, C, C++, Qt, Grails and ActionScript (used in Flex framework of Adobe). I saw Cython project and got really interested in contributing to it. By the way, I saw that the project is trying to participate of GSoC under Python Software Foundation umbrella. I know the student application period have already started, but, I'd really enjoy to participate of GSoC 2011 as a Cython's student. Until day 8 I could work really hard to show you that I can be selected as a GSoC student for Cython. I looked for an Ideas Page of the project but didn't find it, Is there any idea that you have to submit a project in GSoC? If possible, please tell me things that I can start doing to help the project. Best Regards Arthur -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla at molden.no Tue Mar 29 11:45:12 2011 From: sturla at molden.no (Sturla Molden) Date: Tue, 29 Mar 2011 11:45:12 +0200 Subject: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython In-Reply-To: References: <27732698.890.1300864269296.JavaMail.geo-discussion-forums@prfx21> <4D89A4FD.1060408@behnel.de> <4D8C92BA.1020602@molden.no> <4D8C9BCF.8080403@behnel.de> <4D8FBC86.6060906@molden.no> Message-ID: <4D91AA28.3090307@molden.no> Den 29.03.2011 02:09, skrev Robert Bradshaw: > We are very concerned about Python compatibility. I did not intend to say you are not. Judging from Guido's answer to Stephan, I think Guido is worried you are not. And that, BTW, is sufficient to prevent the use of Cython in CPython stdlib. Sturla From vitja.makarov at gmail.com Tue Mar 29 21:11:52 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 29 Mar 2011 23:11:52 +0400 Subject: [Cython] cygdb gdb script problem Message-ID: Running cygdb with Cython installed in the system leads to the following problem: vitja at vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: . Traceback (most recent call last): File "", line 1, in ImportError: No module named Debugger /tmp/tmp1ZvOf9:3: Error in sourced command file: Error while executing Python code. (gdb) So may be it's better to explicitly specify correct path to Cython package in gdb script? Small example patch is attached. -- vitja. -------------- next part -------------- A non-text attachment was scrubbed... Name: cygdb-path-insert.diff Type: text/x-patch Size: 855 bytes Desc: not available URL: From markflorisson88 at gmail.com Tue Mar 29 21:40:22 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 29 Mar 2011 21:40:22 +0200 Subject: [Cython] cygdb gdb script problem In-Reply-To: References: Message-ID: On 29 March 2011 21:11, Vitja Makarov wrote: > Running cygdb with Cython installed in the system leads to the > following problem: > > vitja at vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py > GNU gdb (GDB) 7.2-ubuntu > Copyright (C) 2010 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. ?Type "show copying" > and "show warranty" for details. > This GDB was configured as "x86_64-linux-gnu". > For bug reporting instructions, please see: > . > Traceback (most recent call last): > ?File "", line 1, in > ImportError: No module named Debugger > /tmp/tmp1ZvOf9:3: Error in sourced command file: > Error while executing Python code. > (gdb) > > So may be it's better to explicitly specify correct path to Cython > package in gdb script? > > Small example patch is attached. > > -- > vitja. > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > Hmm, when using cygdb it expects Cython to be in the path. I believe your traceback means that it can find Cython, but not Cython.Debugger, right? Are you sure you're using Python 2.6+? It doesn't work with any version below 2.6 as they don't use the TP_FLAGS_*_SUBCLASS tp_flags. In your patch you probably want to join() the dirname() with os.pardir. However, I don't think it would really solve anything, because if you install Cython, cygdb will be installed in e.g. /usr/bin, so you'd be adding / to the path. If you want to run cygdb from the cython/bin directory, then cython should be listed in PYTHONPATH. From vitja.makarov at gmail.com Tue Mar 29 22:20:14 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 30 Mar 2011 00:20:14 +0400 Subject: [Cython] cygdb gdb script problem In-Reply-To: References: Message-ID: 2011/3/29 mark florisson : > On 29 March 2011 21:11, Vitja Makarov wrote: >> Running cygdb with Cython installed in the system leads to the >> following problem: >> >> vitja at vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py >> GNU gdb (GDB) 7.2-ubuntu >> Copyright (C) 2010 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later >> This is free software: you are free to change and redistribute it. >> There is NO WARRANTY, to the extent permitted by law. ?Type "show copying" >> and "show warranty" for details. >> This GDB was configured as "x86_64-linux-gnu". >> For bug reporting instructions, please see: >> . >> Traceback (most recent call last): >> ?File "", line 1, in >> ImportError: No module named Debugger >> /tmp/tmp1ZvOf9:3: Error in sourced command file: >> Error while executing Python code. >> (gdb) >> >> So may be it's better to explicitly specify correct path to Cython >> package in gdb script? >> >> Small example patch is attached. >> >> -- >> vitja. >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> >> > > Hmm, when using cygdb it expects Cython to be in the path. I believe > your traceback means that it can find Cython, but not Cython.Debugger, > right? Are you sure you're using Python 2.6+? It doesn't work with any > version below 2.6 as they don't use the TP_FLAGS_*_SUBCLASS tp_flags. > That's correct when cygdb is executed it's executed with right cython version, and gdb import system version of cython > In your patch you probably want to join() the dirname() with > os.pardir. However, I don't think it would really solve anything, > because if you install Cython, cygdb will be installed in e.g. > /usr/bin, so you'd be adding / to the path. No it will add /usr/lib to path, path is taken from Cython.Debugger.Cygdb > If you want to run cygdb > from the cython/bin directory, then cython should be listed in > PYTHONPATH. Ok. But you may get in trouble importing system version instead of development one. -- vitja. From markflorisson88 at gmail.com Tue Mar 29 22:46:00 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 29 Mar 2011 22:46:00 +0200 Subject: [Cython] cygdb gdb script problem In-Reply-To: References: Message-ID: On 29 March 2011 22:20, Vitja Makarov wrote: > 2011/3/29 mark florisson : >> On 29 March 2011 21:11, Vitja Makarov wrote: >>> Running cygdb with Cython installed in the system leads to the >>> following problem: >>> >>> vitja at vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py >>> GNU gdb (GDB) 7.2-ubuntu >>> Copyright (C) 2010 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later >>> This is free software: you are free to change and redistribute it. >>> There is NO WARRANTY, to the extent permitted by law. ?Type "show copying" >>> and "show warranty" for details. >>> This GDB was configured as "x86_64-linux-gnu". >>> For bug reporting instructions, please see: >>> . >>> Traceback (most recent call last): >>> ?File "", line 1, in >>> ImportError: No module named Debugger >>> /tmp/tmp1ZvOf9:3: Error in sourced command file: >>> Error while executing Python code. >>> (gdb) >>> >>> So may be it's better to explicitly specify correct path to Cython >>> package in gdb script? >>> >>> Small example patch is attached. >>> >>> -- >>> vitja. >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> >> >> Hmm, when using cygdb it expects Cython to be in the path. I believe >> your traceback means that it can find Cython, but not Cython.Debugger, >> right? Are you sure you're using Python 2.6+? It doesn't work with any >> version below 2.6 as they don't use the TP_FLAGS_*_SUBCLASS tp_flags. >> > > That's correct when cygdb is executed it's executed with right cython version, > and gdb import system version of cython > >> In your patch you probably want to join() the dirname() with >> os.pardir. However, I don't think it would really solve anything, >> because if you install Cython, cygdb will be installed in e.g. >> /usr/bin, so you'd be adding / to the path. > > No it will add /usr/lib to path, path is taken from Cython.Debugger.Cygdb Ah indeed, that code was moved. >> If you want to run cygdb >> from the cython/bin directory, then cython should be listed in >> PYTHONPATH. > > Ok. But you may get in trouble importing system version instead of > development one. Hmm, PYTHONPATH should come before the default search path. But it is true that those eggs can do anything, like insert themselves at the front of the path by putting python code in those .pth files (a gross hack). So indeed, it sounds like a good idea to insert that path at the front of sys.path. > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From dalcinl at gmail.com Wed Mar 30 02:26:53 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 29 Mar 2011 21:26:53 -0300 Subject: [Cython] Cannot profile nogil function. Error or Warn? Message-ID: Error compiling Cython file: ------------------------------------------------------------ ... cdef int PyMPE_Raise(int ierr) except -1 with gil: __Pyx_Raise(RuntimeError, "MPE logging error [code: %d]" % ierr, NULL) return 0 cdef inline int CHKERR(int ierr) nogil except -1: ^ ------------------------------------------------------------ /home/dalcinl/Devel/mpi4py-dev/src/MPE/helpers.pxi:22:5: Cannot profile nogil function. Do we REALLY want this to be an error? Why not just a warning? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From jyf1987 at gmail.com Wed Mar 30 05:33:12 2011 From: jyf1987 at gmail.com (Yunfan Jiang) Date: Wed, 30 Mar 2011 11:33:12 +0800 Subject: [Cython] How to improve the performance when doing string/unicode replace and search? Message-ID: hi, i used to ask some string process question here, and found a bug, it seems you guys fix the bug but not use it and this time , my problem is about the performance, i need to wrote a filter which search sorts of keywords in the target string , and stop if matched, this act require unicode input/output , so i wrote a trie like module to done it, it works ,but i found its too slower than using regex module so could you guys give some tips on string process performance? -- ME = { "name": [ "jyf", "yunfan", "wuxian" ], "im": { "gtalk": "jyf1987 at gmail.com", "msn": "geek42 at live.cn" }, "job": "python engineer", "site": "http://hi.baidu.com/jyf1987", "interested": { "tech": [ "linux", "python", "lua", "php", "html5", "c", "nosql"], "history": ["chinese history", "global history"], "SF": [ "hard SF", "Thought experiment" ], "music": [ "New Age", "Chinese old theme", "Electronic music", "Strange Music :}"] } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Mar 30 07:02:00 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 30 Mar 2011 07:02:00 +0200 Subject: [Cython] How to improve the performance when doing string/unicode replace and search? In-Reply-To: References: Message-ID: <4D92B948.6030301@behnel.de> Yunfan Jiang, 30.03.2011 05:33: > hi, i used to ask some string process question here, and found a bug, it > seems you guys fix the bug but not use it Not sure what you mean by "not use it". > and this time , my problem is about the performance, > i need to wrote a filter which search sorts of keywords in the target > string , and stop if matched, > this act require unicode input/output , so i wrote a trie like module to > done it, it works ,but i found its too slower than using regex module > so could you guys give some tips on string process performance? Note that the right place to ask usage related questions is the Cython users mailing list, not the core developers mailing list. I set a follow-up to point you there. Generally speaking, a trie isn't necessarily fast, and it's certainly not the best algorithmic approach for keyword search. You should read up on Aho-Corasick and friends. I also wrote a simple Cython module that implements a keyword search algorithm ("acora", it's on PyPI), but it's unusable for large sets of keywords due to state explosion. It's pretty fast for smaller sets though. Stefan From jyf1987 at gmail.com Wed Mar 30 08:41:22 2011 From: jyf1987 at gmail.com (Yunfan Jiang) Date: Wed, 30 Mar 2011 14:41:22 +0800 Subject: [Cython] How to improve the performance when doing string/unicode replace and search? In-Reply-To: <4D92B948.6030301@behnel.de> References: <4D92B948.6030301@behnel.de> Message-ID: sorry for ask in the wrong groups, what i mean "not use it" is that, i have download the latest cython source and installed it and i check that the change code is already there, but my bug can still replay On Wed, Mar 30, 2011 at 1:02 PM, Stefan Behnel wrote: > Yunfan Jiang, 30.03.2011 05:33: > > hi, i used to ask some string process question here, and found a bug, it >> seems you guys fix the bug but not use it >> > > Not sure what you mean by "not use it". > > > > and this time , my problem is about the performance, >> i need to wrote a filter which search sorts of keywords in the target >> string , and stop if matched, >> this act require unicode input/output , so i wrote a trie like module to >> done it, it works ,but i found its too slower than using regex module >> so could you guys give some tips on string process performance? >> > > Note that the right place to ask usage related questions is the Cython > users mailing list, not the core developers mailing list. I set a follow-up > to point you there. > > Generally speaking, a trie isn't necessarily fast, and it's certainly not > the best algorithmic approach for keyword search. You should read up on > Aho-Corasick and friends. I also wrote a simple Cython module that > implements a keyword search algorithm ("acora", it's on PyPI), but it's > unusable for large sets of keywords due to state explosion. It's pretty fast > for smaller sets though. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -- ME = { "name": [ "jyf", "yunfan", "wuxian" ], "im": { "gtalk": "jyf1987 at gmail.com", "msn": "geek42 at live.cn" }, "job": "python engineer", "site": "http://hi.baidu.com/jyf1987", "interested": { "tech": [ "linux", "python", "lua", "php", "html5", "c", "nosql"], "history": ["chinese history", "global history"], "SF": [ "hard SF", "Thought experiment" ], "music": [ "New Age", "Chinese old theme", "Electronic music", "Strange Music :}"] } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Wed Mar 30 14:13:47 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 30 Mar 2011 09:13:47 -0300 Subject: [Cython] unspecified return type for extern C++ functions Message-ID: In a previous thread, Robert asked about issuing a warning in the case of unspecified return type for extern C++ functions. I'm definitely +1 for it... Moreover, I would extend it for ANY extern C function. IMHO, this is a case for "explicit is better than implicit". -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169