From stefan_ml at behnel.de Mon Nov 3 11:35:05 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 03 Nov 2014 11:35:05 +0100 Subject: [Cython] Strange bug? with np.int64_t In-Reply-To: References: Message-ID: <54575A59.2010801@behnel.de> Patrick Snape schrieb am 15.10.2014 um 20:27: > To reproduce: > > # distutils: language = c++ > import numpy as np > cimport numpy as np > cimport cython > > > cdef test_np_int64(const np.int64_t test_var): > cdef np.int64_t error = test_var / 2 > > > This fails to compile with the error: > > In function ?const __pyx_t_5numpy_int64_t > __Pyx_div___pyx_t_5numpy_int64_t__const__(__pyx_t_5numpy_int64_t, > __pyx_t_5numpy_int64_t)?: > error: assignment of read-only variable ?q? > q -= ((r != 0) & ((r ^ b) < 0)); > > > Using just a regular int type: > > # distutils: language = c++ > import numpy as np > cimport numpy as np > cimport cython > > > cdef test_int(const int test_var): > cdef int no_error = test_var / 2 > > Works as expected. > > Fixing this is as simple as removing the const that is being complained > about. I assume this is a bug? Yes, looks like a bug to me. Thanks for reporting it. Stefan From zaxebo1 at gmail.com Tue Nov 4 02:08:27 2014 From: zaxebo1 at gmail.com (Zaxebo Yaxebo) Date: Tue, 4 Nov 2014 06:38:27 +0530 Subject: [Cython] [FEATURE REQUEST] VisualBasic "Option Explicit" / Perl "use strict" -- bringing variable pre-declaration to cython optionally Message-ID: Subject: [FEATURE REQUEST] VisualBasic "Option Explicit" / Perl "use strict" -- bringing variable pre-declaration to cython optionally =================================================== Background: already cython supports the following: #program1 a1 = cython.declare(cython.int) # cdef int a1 a2 = cython.declare(cython.double,0.57721) # cdef double a2 =0.57721 #end program1 ==================================================== The New Feature Request: but will there be - a feature in cython, in which - even though we are "not" declaring "data types" of variables, but we are declaring variables themselves. like: ------------ #program2 cython.option_strict(True) ## <------- see this statement a1 = cython.declare(cython.int) # cdef int a1 a2 = cython.declare(cython.double,0.57721) # cdef double a2 =0.57721 a3 = cython.declare(cython.var) #cdef var a3 ## <------- see this statement a4 = cython.declare(cython.var,0.57721) #cdef var a4=0.57721 ## <------- see this statement a3 = 10 # <----- This statement will not give error b1 = 10 # <----- This statement will give warning/error, as variable b1 was not declared and cython.option_strict() is enabled #end program2 --------- Here we have declared the variables a3 and a4 using "cdef var", but we have not declared their data types. For a1 and a2, we have declared their datatypes too. But now, as cython.option_strict() is enabled(i.e., true); HENCE "b1=10" statement will give error when cython compiles, because "b1" variable which was not declared in any sense using "cython.declare(cython.var) or cdef var" . But "a3 = 10" statement in above program will not warning/error,as "a3" is already declared var NOTE: This feature request is analogous to VisualBasic's "Option Explicit" and perl's "use strict" ================== Here are other people also requesting the similar thing: http://stackoverflow.com/questions/613364/is-there-a-need-for-a-use-strict-python-compiler http://stackoverflow.com/questions/13425715/does-python-have-a-use-strict-and-use-warnings-like-in-perl http://stevesprogramming.blogspot.in/2013/03/why-python-needs-perls-use-strict.html http://bytes.com/topic/python/answers/632587-python-feature-request-explicit-variable-declarations Zaxebo1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Wed Nov 5 07:25:42 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 5 Nov 2014 06:25:42 +0000 (UTC) Subject: [Cython] [FEATURE REQUEST] VisualBasic "Option Explicit" / Perl "use strict" -- bringing variable pre-declaration to cython optionally References: Message-ID: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> Zaxebo Yaxebo wrote: > NOTE: This feature request is analogous to VisualBasic's "Option Explicit" > and perl's "use strict" Originally Fortran's "implicit none". I am not sure why you use Python (including Cython) if you prefer to turn off duck typing, though. You know where to find Java or C++ if that is what you want. Sturla From zaxebo1 at gmail.com Wed Nov 5 10:36:56 2014 From: zaxebo1 at gmail.com (Zaxebo Yaxebo) Date: Wed, 5 Nov 2014 09:36:56 +0000 (UTC) Subject: [Cython] [FEATURE REQUEST] VisualBasic References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> Message-ID: Thanks for reply @Sturla. 1) Sturla>>> Originally Fortran's "implicit none". Zaxebo>> Yes, you are right :-) --------------------------------------------------------------- 2) Sturla>>> I am not sure why you use Python (including Cython) if you prefer to turn off duck typing, though. You know where to find Java or C++ if that is what you want. Zaxebo>> I am NOT saying to implement datatyping(that is already part of cython by "cdef int","cdef double"). I am NOT saying to turn off duck typing. What i am requesting is: declaration of variables without data typing (like: cdef "var" ). And "optionally making them to be explicit". It is not required to make each variables data type to be put in the program, only declared as: cdef var a1 So there is no problem to the concept of duck typing in this case. Hence Again , THEIR PRE-DECLARATION (using "var") is to be made explicit, not their datatypes ("int,double" etc do not need to be specified). Hence, we are not making duck typing off. We are still using datatype. --------------------------------------------------------------- 3) If this idea seems acceptable and in congruence with cython philosophy, then I can humbly offer some token amount as sponsor in my individual capacity to the implementor, for this feature to be implemented in cython in a prioritised timeframe. Zaxebo1 From zaxebo1 at gmail.com Wed Nov 5 10:48:22 2014 From: zaxebo1 at gmail.com (Zaxebo Yaxebo) Date: Wed, 5 Nov 2014 09:48:22 +0000 (UTC) Subject: [Cython] [FEATURE REQUEST] VisualBasic References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> Message-ID: ooops,following line in my earlier email: "We are still using datatype." should be read as: "We are still using duck datatyping." Zaxebo1 From sturla.molden at gmail.com Wed Nov 5 17:55:17 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Wed, 5 Nov 2014 16:55:17 +0000 (UTC) Subject: [Cython] [FEATURE REQUEST] VisualBasic References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> Message-ID: <1771173019436899209.764746sturla.molden-gmail.com@news.gmane.org> Zaxebo Yaxebo wrote: > > Zaxebo>> > I am NOT saying to implement datatyping(that is already part of cython by > "cdef int","cdef double"). I am NOT saying to turn off duck typing. > What i am requesting is: declaration of variables without data > typing (like: cdef "var" ). And "optionally making them to be explicit". > It is not required to make each variables data type to be put in the > program, only declared as: > cdef var a1 > So there is no problem to the concept of duck typing in this case. > > Hence Again , THEIR PRE-DECLARATION (using "var") is to be made explicit, > not their datatypes ("int,double" etc do not need to be specified). Hence, > we are not making duck typing off. We are still using datatype. Ok, so it is to guard against spelling errors? Sturla From zaxebo1 at gmail.com Wed Nov 5 19:35:43 2014 From: zaxebo1 at gmail.com (Zaxebo Yaxebo) Date: Wed, 5 Nov 2014 18:35:43 +0000 (UTC) Subject: [Cython] [FEATURE REQUEST] VisualBasic References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> <1771173019436899209.764746sturla.molden-gmail.com@news.gmane.org> Message-ID: Sturla>>> Ok, so it is to guard against spelling errors? Yes, thats a real big problem, once . Others are sufferring from this too, as i mentioned in my original post Here are other people also requesting the similar thing: http://stackoverflow.com/questions/613364/is-there-a-need-for-a-use-strict- python-compiler http://stackoverflow.com/questions/13425715/does-python-have-a-use-strict- and-use-warnings-like-in-perl http://stevesprogramming.blogspot.in/2013/03/why-python-needs-perls-use- strict.html http://bytes.com/topic/python/answers/632587-python-feature-request-explicit- variable-declarations As cython already implements "cdef int a1","cdef double a2" type statements, so implementing "cdef var a1" and optionally making their explicit declaration on, should be a do-able thing. This "explcit declaration of variables - on" can be made by the programmer - either by "cython.option_strict(True)" or by some command line option to cython. ========== As i already mentioned, if this idea is fine with cython community then I am ready to donate some token money for implementation of this feature as soon as possible. Zaxebo1 From stefan_ml at behnel.de Wed Nov 5 20:20:37 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Nov 2014 20:20:37 +0100 Subject: [Cython] [FEATURE REQUEST] VisualBasic In-Reply-To: References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> <1771173019436899209.764746sturla.molden-gmail.com@news.gmane.org> Message-ID: <545A7885.5080905@behnel.de> Zaxebo Yaxebo schrieb am 05.11.2014 um 19:35: > Sturla>>> Ok, so it is to guard against spelling errors? > > Yes, thats a real big problem, once . > > Others are sufferring from this too, as i mentioned in my original post > > Here are other people also requesting the similar thing: > http://stackoverflow.com/questions/613364/is-there-a-need-for-a-use-strict- > python-compiler > http://stackoverflow.com/questions/13425715/does-python-have-a-use-strict- > and-use-warnings-like-in-perl > http://stevesprogramming.blogspot.in/2013/03/why-python-needs-perls-use- > strict.html > http://bytes.com/topic/python/answers/632587-python-feature-request-explicit- > variable-declarations > > As cython already implements "cdef int a1","cdef double a2" type statements, > so implementing "cdef var a1" and optionally making their explicit > declaration on, should be a do-able thing. This has come up a couple of times before, so there are certainly some people who would benefit from a directive that enables warnings specifically about inferred Python object variables. > This "explcit declaration of variables - on" can be made by the programmer - > either by "cython.option_strict(True)" or by some command line option to > cython. A directive, see Options.py. There are already two directives called "warn.undeclared" and "infer_types.verbose". Combining those should get close to what you are looking for. Stefan From zaxebo1 at gmail.com Thu Nov 6 02:35:50 2014 From: zaxebo1 at gmail.com (Zaxebo Yaxebo) Date: Thu, 6 Nov 2014 01:35:50 +0000 (UTC) Subject: [Cython] [FEATURE REQUEST] VisualBasic References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> <1771173019436899209.764746sturla.molden-gmail.com@news.gmane.org> <545A7885.5080905@behnel.de> Message-ID: Stefan>>> A directive, see Options.py. There are already two directives called "warn.undeclared" and "infer_types.verbose". Combining those should get close to what you are looking for. Zaxebo1>> thanks. it works as i wished. You are really really my saviour. =============== Now only thing left, is that I will NOT want to declare the type of each variable as cdef int a1 cdef double a2 I will also want an additional option of "var", so that I can take advantage of DUCK TYPING etc etc, as in: cdef var a3 If this small leftover feature-request of "cdef var" is implemented, then there is nothing like it :-) :-) thanks again Zaxebo1 From stefan_ml at behnel.de Thu Nov 6 07:32:55 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 06 Nov 2014 07:32:55 +0100 Subject: [Cython] [FEATURE REQUEST] VisualBasic In-Reply-To: References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> <1771173019436899209.764746sturla.molden-gmail.com@news.gmane.org> <545A7885.5080905@behnel.de> Message-ID: <545B1617.2040301@behnel.de> Zaxebo Yaxebo schrieb am 06.11.2014 um 02:35: > Stefan>>> A directive, see Options.py. There are already two directives > called "warn.undeclared" and "infer_types.verbose". Combining those should > get close to what you are looking for. > > Zaxebo1>> thanks. it works as i wished. You are really really my saviour. > > =============== > Now only thing left, is that I will NOT want to declare the type of each > variable as > cdef int a1 > cdef double a2 > I will also want an additional option of "var", so that I can take advantage > of DUCK TYPING etc etc, as in: > cdef var a3 Do you mean like cdef object a3 or cdef a3 ? Because that already exists and assigns type "arbitrary Python object" to the variable. If you want to be more specific, use fused types (a.k.a. generics). Stefan From robertwb at gmail.com Thu Nov 6 08:34:21 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 5 Nov 2014 23:34:21 -0800 Subject: [Cython] New function (pointer) syntax. Message-ID: I'd like to propose a more pythonic way to declare function pointer types, namelye type0 (*[ident])(type1, type2, type3) would instead become (type1, type2, type3) -> type0 [ident] I have a pull request up at https://github.com/cython/cython/pull/333; what do people think? - Robert From stefan_ml at behnel.de Thu Nov 6 09:29:45 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 06 Nov 2014 09:29:45 +0100 Subject: [Cython] New function (pointer) syntax. In-Reply-To: References: Message-ID: <545B3179.7010802@behnel.de> Robert Bradshaw schrieb am 06.11.2014 um 08:34: > I'd like to propose a more pythonic way to declare function pointer > types, namelye > > type0 (*[ident])(type1, type2, type3) > > would instead become > > (type1, type2, type3) -> type0 [ident] Not convinced. Looks quite magic, very different from the C that people would know, and takes a while reading from left to right until you see the "->" which is essentially the only indication of what is happening here. Also, "->" has a widely known meaning in C/C++ which makes the above very confusing. I think in C when thinking about function pointers, and there is no "def ..." keyword indication here that would switch me back to Python (as for signature annotations). I do see the problem of "where did I have to put that '*' again in the C declaration?", but this looks like it will produce more confusion than it avoids. Many (or most?) function pointer type declarations will come from some C library's header file, so it means making it more difficult for users to copy over the declaration. Stefan From zaxebo1 at gmail.com Thu Nov 6 09:49:07 2014 From: zaxebo1 at gmail.com (Zaxebo Yaxebo) Date: Thu, 6 Nov 2014 08:49:07 +0000 (UTC) Subject: [Cython] [FEATURE REQUEST] VisualBasic References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> <1771173019436899209.764746sturla.molden-gmail.com@news.gmane.org> <545A7885.5080905@behnel.de> <545B1617.2040301@behnel.de> Message-ID: o good, that solved this issue. Thanks a lot Zaxebo1 From stefan_ml at behnel.de Thu Nov 6 17:57:41 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 06 Nov 2014 17:57:41 +0100 Subject: [Cython] [FEATURE REQUEST] VisualBasic In-Reply-To: References: <921318296436861358.418908sturla.molden-gmail.com@news.gmane.org> <1771173019436899209.764746sturla.molden-gmail.com@news.gmane.org> <545A7885.5080905@behnel.de> <545B1617.2040301@behnel.de> Message-ID: <545BA885.6030602@behnel.de> Zaxebo Yaxebo schrieb am 06.11.2014 um 09:49: > o good, that solved this issue. > > Thanks a lot If you think there's anything missing from the documentation that would have helped you find these features yourself, please consider submitting a pull request that improves the current state. Stefan From robertwb at gmail.com Thu Nov 6 18:15:12 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 6 Nov 2014 09:15:12 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <545B3179.7010802@behnel.de> References: <545B3179.7010802@behnel.de> Message-ID: On Thu, Nov 6, 2014 at 12:29 AM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 06.11.2014 um 08:34: >> I'd like to propose a more pythonic way to declare function pointer >> types, namelye >> >> type0 (*[ident])(type1, type2, type3) >> >> would instead become >> >> (type1, type2, type3) -> type0 [ident] > > Not convinced. Looks quite magic, very different from the C that people > would know, and takes a while reading from left to right until you see the > "->" which is essentially the only indication of what is happening here. I suppose it depends on whether one knows just C, is familiar with a wide variety of other (more functional) languages. I think it's a lot easier for people to read who only know Python, or just a smattering of C. > Also, "->" has a widely known meaning in C/C++ which makes the above very > confusing. I think in C when thinking about function pointers, and there is > no "def ..." keyword indication here that would switch me back to Python > (as for signature annotations). > > I do see the problem of "where did I have to put that '*' again in the C > declaration?", but this looks like it will produce more confusion than it > avoids. Hmm, I hadn't thought of that meaning of ->. I don't think it would be as confusing in context, which will be immediately following cdef or in a function argument, e.g. one would write cdef (float -> float, float, float) -> float integrator = get_integrator(...) which doesn't to me look like member access, and is much clear than cdef float (*integrator)(float (*)(float), float, float) = get_integrator(...) This becomes especially clear for return types, e.g. cdef ((float -> float, float, float) -> float) get_integrator(char* algorithm): if algorithm == ... vs cdef float (*get_integrator(char* algorithm))(float (*)(float), float, float): if algorithm == ... > Many (or most?) function pointer type declarations will come from some C > library's header file, so it means making it more difficult for users to > copy over the declaration. I really hope, in the long run, people won't have to manually do these declarations. As well as making things easier to read, part of the motivation is to come up with a syntax for declaring this type without having to add the notion of declarators (empty or not) to the grammar. Essentially everyone thinks "cdef type var" even though that's not currently the true grammar. Another option that just occurred to me would be the syntax "lambda type1, type2, type3: type0" so for this example one would have cdef lambda (lambda float: float), float, float: float integrator = get_integrator(...) doesn't read as nice to me, but thought I'd throw it out there. - Robert From stefan_ml at behnel.de Thu Nov 6 18:59:38 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 06 Nov 2014 18:59:38 +0100 Subject: [Cython] New function (pointer) syntax. In-Reply-To: References: <545B3179.7010802@behnel.de> Message-ID: <545BB70A.5030900@behnel.de> Robert Bradshaw schrieb am 06.11.2014 um 18:15: > On Thu, Nov 6, 2014 at 12:29 AM, Stefan Behnel wrote: >> Robert Bradshaw schrieb am 06.11.2014 um 08:34: >>> I'd like to propose a more pythonic way to declare function pointer >>> types, namelye >>> >>> type0 (*[ident])(type1, type2, type3) >>> >>> would instead become >>> >>> (type1, type2, type3) -> type0 [ident] >> >> Not convinced. Looks quite magic, very different from the C that people >> would know, and takes a while reading from left to right until you see the >> "->" which is essentially the only indication of what is happening here. > > I suppose it depends on whether one knows just C, is familiar with a > wide variety of other (more functional) languages. I think it's a lot > easier for people to read who only know Python, or just a smattering > of C. The thing is, in order to use Cython, you currently only need to understand Python, some C/C++ and a minor bit of special Cython syntax. The more new syntax we add outside of Python/C/C++, the higher the entry bar is raised for both the "mostly Python" and "I know C" users. >> Also, "->" has a widely known meaning in C/C++ which makes the above very >> confusing. I think in C when thinking about function pointers, and there is >> no "def ..." keyword indication here that would switch me back to Python >> (as for signature annotations). >> >> I do see the problem of "where did I have to put that '*' again in the C >> declaration?", but this looks like it will produce more confusion than it >> avoids. > > Hmm, I hadn't thought of that meaning of ->. I don't think it would be > as confusing in context, which will be immediately following cdef or > in a function argument, e.g. one would write > > cdef (float -> float, float, float) -> float integrator = > get_integrator(...) My brain can't even parse that. Does it join as (float -> (float, float, float)) or ((float -> float), float, float) ? A ctypedef would definitely help here. > which doesn't to me look like member access, and is much clear than > > cdef float (*integrator)(float (*)(float), float, float) = > get_integrator(...) Apart from the (*), at least this makes it clear what gets passed into what. > This becomes especially clear for return types, e.g. > > cdef ((float -> float, float, float) -> float) > get_integrator(char* algorithm): > if algorithm == ... > > vs > > cdef float (*get_integrator(char* algorithm))(float (*)(float), > float, float): > if algorithm == ... Same thing as above. The nesting is clearer in the latter. >> Many (or most?) function pointer type declarations will come from some C >> library's header file, so it means making it more difficult for users to >> copy over the declaration. > > I really hope, in the long run, people won't have to manually do these > declarations. They'll definitely stay. At least as an advanced feature and for the simple (entry) cases, they provide both an easily understandable mapping and a handy way to tune the declarations to make their usage friendlier in Cython code. > As well as making things easier to read, part of the motivation is to > come up with a syntax for declaring this type without having to add > the notion of declarators (empty or not) to the grammar. Essentially > everyone thinks "cdef type var" even though that's not currently the > true grammar. True. But why not require at least a ctypedef for these things? Ad-hoc declaration of C function types is just bound to get in the way of readability. > Another option that just occurred to me would be the syntax "lambda > type1, type2, type3: type0" so for this example one would have > > cdef lambda (lambda float: float), float, float: float integrator > = get_integrator(...) > > doesn't read as nice to me, but thought I'd throw it out there. It's funny that lambda uses the same expression separator as the signature annotations in Py3. Hadn't even noticed that before. Looks a bit funny at first sight, but definitely clearer than any of the above. Stefan From robertwb at gmail.com Thu Nov 6 19:56:49 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 6 Nov 2014 10:56:49 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <545BB70A.5030900@behnel.de> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> Message-ID: [Cc'ing elsewhere for more feedback. Also top-posting for initial impressions before the discussion.] Here's some proposed function pointer syntaxes; which are the most obvious to understand/read/remember? Can you figure them out? cdef float (*F)(float) cdef float (*G)(float (*)(float), float, float) cdef float ((*H)(char*))(float (*)(float), float, float) vs cdef float -> float F cdef (float -> float, float, float) -> float G cdef (char*) -> (float -> float, float, float) -> float H vs cdef lambda float: float F cdef lambda (lambda float: float), float, float: float G cdef lambda (char*): lambda: (lambda float: float), float, float: float H If you want a hint, the last is something that returns numerical integration algorithm given a string name. Yes, you could use typedefs, but you shouldn't have to. especially for the first. On Thu, Nov 6, 2014 at 9:59 AM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 06.11.2014 um 18:15: >> On Thu, Nov 6, 2014 at 12:29 AM, Stefan Behnel wrote: >>> Robert Bradshaw schrieb am 06.11.2014 um 08:34: >>>> I'd like to propose a more pythonic way to declare function pointer >>>> types, namelye >>>> >>>> type0 (*[ident])(type1, type2, type3) >>>> >>>> would instead become >>>> >>>> (type1, type2, type3) -> type0 [ident] >>> >>> Not convinced. Looks quite magic, very different from the C that people >>> would know, and takes a while reading from left to right until you see the >>> "->" which is essentially the only indication of what is happening here. >> >> I suppose it depends on whether one knows just C, is familiar with a >> wide variety of other (more functional) languages. I think it's a lot >> easier for people to read who only know Python, or just a smattering >> of C. > > The thing is, in order to use Cython, you currently only need to understand > Python, some C/C++ and a minor bit of special Cython syntax. The more new > syntax we add outside of Python/C/C++, the higher the entry bar is raised > for both the "mostly Python" and "I know C" users. True. However, " -> return_annotation" is still Python syntax. >>> Also, "->" has a widely known meaning in C/C++ which makes the above very >>> confusing. I think in C when thinking about function pointers, and there is >>> no "def ..." keyword indication here that would switch me back to Python >>> (as for signature annotations). >>> >>> I do see the problem of "where did I have to put that '*' again in the C >>> declaration?", but this looks like it will produce more confusion than it >>> avoids. >> >> Hmm, I hadn't thought of that meaning of ->. I don't think it would be >> as confusing in context, which will be immediately following cdef or >> in a function argument, e.g. one would write >> >> cdef (float -> float, float, float) -> float integrator = >> get_integrator(...) > > My brain can't even parse that. Does it join as > > (float -> (float, float, float)) > > or > > ((float -> float), float, float) > > ? The comma token remains lower precedence than about everything else. It's just like a parameter list cdef integrate(float -> float f, float a, float b): ... f is something that takes floats to floats. > A ctypedef would definitely help here. > > >> which doesn't to me look like member access, and is much clear than >> >> cdef float (*integrator)(float (*)(float), float, float) = >> get_integrator(...) > > Apart from the (*), at least this makes it clear what gets passed into what. > > >> This becomes especially clear for return types, e.g. >> >> cdef ((float -> float, float, float) -> float) >> get_integrator(char* algorithm): >> if algorithm == ... >> >> vs >> >> cdef float (*get_integrator(char* algorithm))(float (*)(float), >> float, float): >> if algorithm == ... > > Same thing as above. The nesting is clearer in the latter. Clearly we think differently... I can pick apart the latter definition, but I have to admit it took a lot longer to type than the former, and I still see "cdef float blah(...., float, float): ..." when in fact I'm declaring something that takes a char* (or should that read, "takes something that dereferences to a char" :-)). I am curious, when you read "cdef int * p" do you parse this as "cdef (int*) p" or "cdef int (*p)" 'cause for me no matter how well I know it's the latter, I think the former (i.e. I think "I'm declaring a variable of type p that's of int pointer type.") >>> Many (or most?) function pointer type declarations will come from some C >>> library's header file, so it means making it more difficult for users to >>> copy over the declaration. >> >> I really hope, in the long run, people won't have to manually do these >> declarations. > > They'll definitely stay. At least as an advanced feature and for the simple > (entry) cases, they provide both an easily understandable mapping and a > handy way to tune the declarations to make their usage friendlier in Cython > code. I should have said they'll be less common, at least for the "copy header file" in which I'd rather point to the header file and list members to declare. For advance use (e.g. you want to mis-declare types) we'd still need this. >> As well as making things easier to read, part of the motivation is to >> come up with a syntax for declaring this type without having to add >> the notion of declarators (empty or not) to the grammar. Essentially >> everyone thinks "cdef type var" even though that's not currently the >> true grammar. > > True. But why not require at least a ctypedef for these things? Ad-hoc > declaration of C function types is just bound to get in the way of readability. Not being able to declare this without a ctypedef would be a weakness in the language, especially as ctypedefs can't be declared locally. Also, it just defers the ugliness to elsewhere, e.g. ctypedef float (*FloatFunc)(float) ctypedef float (*IntegrateFunc)(FloatFunc, float, float) The reason ctypedefs help, and are so commonly used for with function pointers, is because the existing syntax is just so horrendously bad. If there's a clear way to declare a function taking a single float and returning a single float, no typedef needed. >> Another option that just occurred to me would be the syntax "lambda >> type1, type2, type3: type0" so for this example one would have >> >> cdef lambda (lambda float: float), float, float: float integrator >> = get_integrator(...) >> >> doesn't read as nice to me, but thought I'd throw it out there. > > It's funny that lambda uses the same expression separator as the signature > annotations in Py3. Hadn't even noticed that before. > > Looks a bit funny at first sight, but definitely clearer than any of the above. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Thu Nov 6 20:56:41 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 06 Nov 2014 20:56:41 +0100 Subject: [Cython] New function (pointer) syntax. In-Reply-To: References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> Message-ID: <545BD279.5060203@behnel.de> Robert Bradshaw schrieb am 06.11.2014 um 19:56: > On Thu, Nov 6, 2014 at 9:59 AM, Stefan Behnel wrote: >> Robert Bradshaw schrieb am 06.11.2014 um 18:15: >>> This becomes especially clear for return types, e.g. >>> >>> cdef ((float -> float, float, float) -> float) >>> get_integrator(char* algorithm): >>> if algorithm == ... >>> >>> vs >>> >>> cdef float (*get_integrator(char* algorithm))(float (*)(float), >>> float, float): >>> if algorithm == ... >> >> Same thing as above. The nesting is clearer in the latter. > > Clearly we think differently... I can pick apart the latter > definition, but I have to admit it took a lot longer to type than the > former, and I still see "cdef float blah(...., float, float): ..." > when in fact I'm declaring something that takes a char* (or should > that read, "takes something that dereferences to a char" :-)). I already said that C function types have their own quirks - I keep writing (func*) instead of (*func), for example. It feels just arbitrary and fails to stick in my head. But "->" is too well known as an operator when it comes to pointers (which function types are in C) to provide for a *more* readable alternative. Just because some functional languages use it in some specific way doesn't mean it fits into a language that mixes Python and C. > I am curious, when you read "cdef int * p" do you parse this as "cdef > (int*) p" or "cdef int (*p)" Definitely the former. And the complexity that some type declarations can take in C (let alone C++) is just devastating. Problem is, any new syntax in Cython will have to co-exist with the existing (C) syntax potentially forever, and will add to the language learning curve. People will learn one syntax, but have to understand the second syntax in order to read other people's code. And others will start adopting the new syntax in code that uses the old syntax without cleaning up first. If we go that route of adding yet another syntax, it needs to be an obvious one from both a "C with Python features and syntax" and a "Python with C datatypes" point of view. In short, it has to fit into the existing language mix. Stefan From robertwb at gmail.com Thu Nov 6 21:39:03 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 6 Nov 2014 12:39:03 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <545BD279.5060203@behnel.de> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <545BD279.5060203@behnel.de> Message-ID: On Thu, Nov 6, 2014 at 11:56 AM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 06.11.2014 um 19:56: >> On Thu, Nov 6, 2014 at 9:59 AM, Stefan Behnel wrote: >>> Robert Bradshaw schrieb am 06.11.2014 um 18:15: >>>> This becomes especially clear for return types, e.g. >>>> >>>> cdef ((float -> float, float, float) -> float) >>>> get_integrator(char* algorithm): >>>> if algorithm == ... >>>> >>>> vs >>>> >>>> cdef float (*get_integrator(char* algorithm))(float (*)(float), >>>> float, float): >>>> if algorithm == ... >>> >>> Same thing as above. The nesting is clearer in the latter. >> >> Clearly we think differently... I can pick apart the latter >> definition, but I have to admit it took a lot longer to type than the >> former, and I still see "cdef float blah(...., float, float): ..." >> when in fact I'm declaring something that takes a char* (or should >> that read, "takes something that dereferences to a char" :-)). > > I already said that C function types have their own quirks - I keep writing > (func*) instead of (*func), for example. It feels just arbitrary and fails > to stick in my head. But "->" is too well known as an operator when it > comes to pointers (which function types are in C) to provide for a *more* > readable alternative. Just because some functional languages use it in some > specific way doesn't mean it fits into a language that mixes Python and C. Even Java8 is adopting this token for lambdas, though I have to admit my bias probably comes at least partially from mathematics (where I write "let f: Z -> Z be a map from the integers to itself")... and I was happy that def foo(a: x, b: y) -> z: ... simplified nicely down to (x, y) -> z but I see your point that this is much more foreign coming from the C side of things. >> I am curious, when you read "cdef int * p" do you parse this as "cdef >> (int*) p" or "cdef int (*p)" > > Definitely the former. And the complexity that some type declarations can > take in C (let alone C++) is just devastating. > > Problem is, any new syntax in Cython will have to co-exist with the > existing (C) syntax potentially forever, and will add to the language > learning curve. People will learn one syntax, but have to understand the > second syntax in order to read other people's code. And others will start > adopting the new syntax in code that uses the old syntax without cleaning > up first. I wouldn't say forever. Yes, this sounds a bit like Python 2 vs. 3, but we could provide a tool to do the conversion (completely correctly in this case) and as developers often run Cython themselves and ship the .c files it should be a much less painful transition. > If we go that route of adding yet another syntax, it needs to be an obvious > one from both a "C with Python features and syntax" and a "Python with C > datatypes" point of view. In short, it has to fit into the existing > language mix. +1. I hope to get more than two pairs of eyeballs on the issue. - Robert From greg.ewing at canterbury.ac.nz Thu Nov 6 23:46:53 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Nov 2014 11:46:53 +1300 Subject: [Cython] New function (pointer) syntax. In-Reply-To: References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> Message-ID: <545BFA5D.30405@canterbury.ac.nz> Robert Bradshaw wrote: > If you want a hint, the last is something that returns numerical > integration algorithm given a string name. Yes, you could use > typedefs, but you shouldn't have to. I don't find *any* of those particularly easy to read in the third case, or even the second. Using typedefs to make the intention clear, even if the syntax doesn't require it, seems entirely appropriate to me. > I am curious, when you read "cdef int * p" do you parse this as "cdef > (int*) p" or "cdef int (*p)" C syntax is sufficiently well embedded in my brain that I parse it as "int (*p)", and read it as "p points to an int" or "p is of type pointer-to-int". But I don't find myself needing to parse "int * p" very often, because I always write it as "int *p", and so do most other people who write C. Whenever I see something like "int * p" or "int* p", it makes me think it was written by someone who doesn't understand C very well. -- Greg From robertwb at gmail.com Thu Nov 6 23:52:33 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 6 Nov 2014 14:52:33 -0800 Subject: [Cython] [cython-users] Re: New function (pointer) syntax. In-Reply-To: References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> Message-ID: On Thu, Nov 6, 2014 at 1:07 PM, Nils Bruin wrote: > On Thursday, November 6, 2014 10:57:23 AM UTC-8, Robert Bradshaw wrote: >> >> cdef float -> float F >> cdef (float -> float, float, float) -> float G >> cdef (char*) -> (float -> float, float, float) -> float H > > Is there any precedent for an infix operator in type declarations? I find it > really surprising. The "input -> output" syntax is used for anonymous functions and callable types in several languages, from Haskell to OCaml to Java. Python 3 introduces -> to annotate function return types in PEP 3107. > It's a little better with parentheses I think: > > cdef (float -> float) F > > Without them, the "float F" seems to visually bind much more closely than > the "float -> float", contrary to your proposed precedence. Yeah, the parentheses make this more clear, though cdef float -> (float F) doesn't really have a reasonable interpretation. The syntax "[type] [space] [name]" is a bit unfortunate--we already have ugly, special logic to handle things like "cdef unsigned short int foo" or even worse "def foo(self, int): ..." > Also, C's > convention that "a->b == *a.b" makes the "->" use here unexpected, > especially because it's used in an environment where pointer (type)s are > involved. Yeah, Stefan pointed this out too, though we don't use -> in Cython (except for function return annotation). I think the fact that these are function *pointers* wouldn't need to be called out, given that functions can be assigned to function pointers and function pointers are callable just like functions. They'd behave more like first class function objects. >> cdef lambda float: float F >> cdef lambda (lambda float: float), float, float: float G >> cdef lambda (char*): lambda: (lambda float: float), float, float: >> float H > > > At least this overcomes the infix issue a bit: right from the start, the > "lambda" hints that this is a function type. However, visually again this > reads to me (because : in normal writing a : would separate sentences of > words separated by spaces, hence indicating that a space has higher > precedence than a colon) as: > > cdef lambda float: (float F) > > which can be circumvented by parentheses: > > cdef (lambda float: float) F > > The "lambda" solution doesn't look very appealing here, but I think (with > parentheses) it is the one that is most likely to be unambiguously clear. Yeah, it looks funny, but is harder to miss-interpret. - Robert From robertwb at math.washington.edu Fri Nov 7 06:04:28 2014 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 6 Nov 2014 21:04:28 -0800 Subject: [Cython] [sage-devel] Re: New function (pointer) syntax. In-Reply-To: <87636c75-c073-4d34-9537-cd5a15ddf7fe@googlegroups.com> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <87636c75-c073-4d34-9537-cd5a15ddf7fe@googlegroups.com> Message-ID: Thanks for all the feedback! On Thu, Nov 6, 2014 at 7:13 PM, Travis Scrimshaw wrote: > Here's my 2 cents. >> >> >> cdef float (*F)(float) >> cdef float (*G)(float (*)(float), float, float) >> cdef float ((*H)(char*))(float (*)(float), float, float) > > > I prefer this one because it mimics C function pointers (at least how I > remember, I'm somewhat rusty). It is C function pointer syntax... if you consider that a plus (aside from compatibility). > When I saw the "->", I thought of pointer > redirection (without reading the rest of the post which mentioned it) and > believe it could be a major source of confusion. I'm surprised so many people are noting this; good feedback. > However since cython is > suppose to mimic python as much as possible (please correct me if I'm wrong > here), perhaps the lambda one is better? Yeah, it's a bit odd, but less confusing. A goal is a syntax such that when you look at it you know what it means, unlike, say, the C syntax that you have to learn how to decode. When I first saw it in Miranda ages ago, but clearly others think of C first when they think of something of type "float -> float." On Thu, Nov 6, 2014 at 8:06 PM, Peter Schay wrote: > > The C function pointer syntax is clever and makes a lot of sense but > reading/writing a complicated declaration usually feels like I'm solving a > puzzle. Exactly what I want to avoid. Certainly violates several points of the Zen of Python. > IANACG (I am not a compiler guy) so I am sure there must be some flaw in the > following but I can't help asking whether there would be anything wrong with > this form, which reads more like a sentence ("a pointer to a function F with > args x which returns y"): > > cdef lambda F(float): float > cdef lambda G( > (lambda (float): float), > float, > float > ): float > cdef lambda H(char *): (lambda (float, float, float): float) In Python "lambda (a, b): ..." with the parentheses means a function taking one argument that's unpacked into two variables. > One could also use an underscore for the unnamed lambdas, maybe. One of the main goals is to move away from C style declarators, i.e. actually have "cdef [type] [name]" instead of "cdef [basetype] [mix_of_names_and_type_info]. So one would write cdef (float -> float) f, g, h to declare three variables of type float -> float. Then we might have hope of an actual, comprehensible grammar for Cython. (Few folks think in terms of declarators, so one would write "int[10] x" to declare an array of size 10 rather than "int x[10].") On Thu, Nov 6, 2014 at 8:46 PM, Nathann Cohen wrote: > Hello ! > > To me the second looks the most clear. Except that I would have written > > cdef (char*) -> ((float -> float, float, float) -> float H) instead of > > cdef (char*) -> (float -> float, float, float) -> float H > > But well, one gets used to things like that. Yeah, cdef ( (char*) -> ((float -> float, float, float) -> float) ) H would be just fine as well for those who like parentheses. > It is clearer but indeed is it surprising to find such a syntax in Cython... > Actually like Travis I would expect Cython to support the first, even though > it is the least obvious... It does now. > If both are implemented I will definitele use the second whenever I can :-) Me too. On Thu, Nov 6, 2014 at 8:48 PM, 1989lzhh <1989lzhh at gmail.com> wrote: > > Here are numba kind function annotation, I guess it may fit in here. > cdef float(float) F > cdef float(float(float), float, float) G > cdef float(float(float), float, float)(char*) H > I personally feel this kind of annotation is more packed that using ->. That's an interesting alternative. Would float(float)* be a pointer to a function pointer then? (float*)(float) something that returned a pointer to a float? - Robert From stefan_ml at behnel.de Fri Nov 7 08:18:35 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 07 Nov 2014 08:18:35 +0100 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> Message-ID: <545C724B.8080403@behnel.de> 1989lzhh schrieb am 07.11.2014 um 05:48: >> ? Nov 7, 2014?02:56?Robert Bradshaw ??? >> Here's some proposed function pointer syntaxes; which are the most >> obvious to understand/read/remember? Can you figure them out? >> >> cdef float (*F)(float) >> cdef float (*G)(float (*)(float), float, float) >> cdef float ((*H)(char*))(float (*)(float), float, float) >> >> vs >> >> cdef float -> float F >> cdef (float -> float, float, float) -> float G >> cdef (char*) -> (float -> float, float, float) -> float H >> >> vs >> >> cdef lambda float: float F >> cdef lambda (lambda float: float), float, float: float G >> cdef lambda (char*): lambda: (lambda float: float), float, float: float H >> >> >> If you want a hint, the last is something that returns numerical >> integration algorithm given a string name. Yes, you could use >> typedefs, but you shouldn't have to. especially for the first. >> > Here are numba kind function annotation, I guess it may fit in here. > cdef float(float) F > cdef float(float(float), float, float) G > cdef float(float(float), float, float)(char*) H > I personally feel this kind of annotation is more packed that using ->. I find it clearer than any of the above. And it still allows the use of named arguments, i.e. you could say cdef char*(float x, int y) F = ... F(y=1, x=2.0) Drawback is that you have to parse up to the name in order to distinguish it from cdef char*(float x, int y): ... Then there's also cdef char*(float x, int y) nogil F And we have to disallow cdef char*(float x, int y) with gil F It gets a bit less readable when you take a proper lower-case variable name, e.g. cdef char*(float x, int y) nogil get_gil We could fix some of that by allowing cdef nogil: cdef char*(float x, int y) get_gil But then, that's adding yet another special case. Stefan From cblake at pdos.csail.mit.edu Thu Nov 6 22:35:36 2014 From: cblake at pdos.csail.mit.edu (C Blake) Date: Thu, 6 Nov 2014 16:35:36 -0500 Subject: [Cython] New function (pointer) syntax. In-Reply-To: Message-ID: <20141106213536.GA30041@pdos.lcs.mit.edu> I think you should just use the C declarator syntax. Cython already allows you to say "cdef int *foo[10]". Declarators aren't bad - just poorly taught, though I can see some saying those are the same thing. More below. I absolutely like the declarator one the most, and the lambda one second most. Declarator style makes it far easier to take C code/.h files using function pointers over to Cython. So, this discussion also depends on whether you view Cython as a bridge to C libs or its own island/bias toward pure Py3. One other proposal that might appease Stefan's "Python lexical cues" where he was missing "def" would be to take the Python3 function definition header syntax and strip just variable names. I.e., keep the ":"s def foo(x: type1, y: type2, z: type3) -> type0: pass goes to (: type1, : type2, : type3) -> type0 [ident] I don't think that looks like anything else that might valid in C or Python. It does just what C does - strip variable names from a function header, but the ":"s maybe key your brain into a different syntax mode since they are arguably more rare in C. (Besides stripping names, C has some extra parens for precedence/associativity - which admittedly are tuned to make use-expressions simpler than type-expressions.) Anyway, I don't really like my own proposal above. Just mentioning it for completeness in case the ":"s help anyone. Robert wrote: >I really hope, in the long run, people won't have to manually do these >declarations. I agree they'll always be there somehow and also with Stefan's comments about the entry bar. So, most people not needing them most of the time doesn't remove the question. >I am curious, when you read "cdef int * p" do you parse this as "cdef >(int*) p" or "cdef int (*p)" 'cause for me no matter how well I know >it's the latter, I think the former (i.e. I think "I'm declaring a >variable of type p that's of int pointer type.") >[..] >Essentially everyone thinks "cdef type var" even though that's not >currently the true grammar. >[..] >The reason ctypedefs help, and are so commonly used for with function >pointers, is because the existing syntax is just so horrendously bad. >If there's a clear way to declare a function taking a single float and >returning a single float, no typedef needed. No, no, no. Look, generations of C/C++ programmers have been done monumental disservice by textbooks/code/style guides that suggest "int* p" is *any less* confusing than spacing "2+3/x" as "2+3 / x". Early on in my C exposure someone pointed this out and I've never been confused since. It's a syntax-semantics confusion. Concrete syntax has always been right associative dereference *. In this syntax family, the moment any operators []/*/() are introduced, you have to start formatting it/thinking of it as a real expression, and that formatting should track the syntax not semantics like in your head "pointer to"/indirection speak or whatever. Spacing it as if * were left associative to the type name is misleading at best. If you can only think of type decls in general as (type, var) pairs syntactically and semantically then *of course* you find typedefs more clear. They make pairing more explicit & shrink the expression tree to be more trivial. You should *still* space the typedef itself in a way suggestive of the actual concrete syntax -- "typedef int *p" (or "ctypedef") just like you shouldn't write "2+3 / x". You should still not essentially think of "ctypedef type var" *either*, but rather "typedef basetype expr". In short, "essentially everyone" *should* think and be taught and have it reinforced and "gel" by spacing that "basetype expr" is the syntax to create a type-var bindings semantically, and only perceive "type var" as just one simple case. Taking significance of space in Python/Cython one step further, "int* p" could even be a hard syntax error, but I'm not seriously proposing that change. I really do not think it is "essentially everyone". You know better as you said anyway, but are in conflict with yourself, I think syntax-semantics wise. Semantically, pointer indirection loads from an address before using, and sure that can be confusing to new programmers in its own right. Trying to unravel that confusion with anti-syntax spacing/thought cascades the wrong way out of the whole situation and contributes to your blocked assimilation. *If* the space guides you or barring space parens guide you, you quickly get to never forgetting that types are inside-out/inverse/what-I-get-if specifications. Note that this is as it would if +,/ had somehow tricky concepts somehow "fixable" by writing "2+3 / x" all the time. Arithmetic isn't a binding..so the analogy is hard to complete, but my point is ad nauseum at this stage (or even sooner! ;-). Undermine syntax with contrary whitespace and of course it will seem bad/be harder. It might even lock you in to thought patterns that make it really hard to think about it how you know you "ought" to. Anyway, more the point, Cython has "cdef" not "py3def" or "javadef" or whatever, after all. There is even the cdef: blocks where all the decls and inits look eminently C-like. Having an alternative ptr syntax from C only for functions but not arrays seems wrong to me *unless* it's really part of a general Py3 annotations syntax move. Stefan was in some mypy-in-Py3 thread last Summer. If the situation were Cython just moving that way to be a Py3-with-mypy compiler with some extras for C integration, that would be a different story to me. Then trying to make function pointers seem like definitions sans-names might make more sense. It would really ease moves to Cython from pure Py3 users used to the type annotations "for checking" to apply to code generation. That may be such a big move from Cython now that it might almost deserve some "Cython3" fork that could also drop other redundant things (like pre-memory views NumPy integration that seems not as good as more general memviews or something). All that being said, using lambda syntax to signify types as well as values seems an interesting idea, if it would really work. But we know C declarators will work for all things C. I don't see why we shouldn't just stick to that. Right now and with declarators you can tell people you just need to learn C decl syntax and Python. That's a VERY simple thing to say (even if it requires some learning work, and even if it isn't 100% exact). There is a lot of documentation out there to learn C decls. From stefan_ml at behnel.de Fri Nov 7 08:26:53 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 07 Nov 2014 08:26:53 +0100 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <545C724B.8080403@behnel.de> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> <545C724B.8080403@behnel.de> Message-ID: <545C743D.20303@behnel.de> Stefan Behnel schrieb am 07.11.2014 um 08:18: > We could fix some of that by allowing > > cdef nogil: > cdef char*(float x, int y) get_gil > > But then, that's adding yet another special case. ... except when it's already supported, which it turns out to be... Stefan From insertinterestingnamehere at gmail.com Fri Nov 7 08:30:47 2014 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Thu, 6 Nov 2014 23:30:47 -0800 (PST) Subject: [Cython] [cython-users] Re: New function (pointer) syntax. In-Reply-To: <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> Message-ID: <53004edb-16d8-4ebb-aab2-22e1248f3ec8@googlegroups.com> On Thursday, November 6, 2014 9:48:53 PM UTC-7, 1989lzhh wrote: > > > > > > ? Nov 7, 2014?02:56?Robert Bradshaw > > ??? > > > > [Cc'ing elsewhere for more feedback. Also top-posting for initial > > impressions before the discussion.] > > > > Here's some proposed function pointer syntaxes; which are the most > > obvious to understand/read/remember? Can you figure them out? > > > > cdef float (*F)(float) > > cdef float (*G)(float (*)(float), float, float) > > cdef float ((*H)(char*))(float (*)(float), float, float) > > > > vs > > > > cdef float -> float F > > cdef (float -> float, float, float) -> float G > > cdef (char*) -> (float -> float, float, float) -> float H > > > > vs > > > > cdef lambda float: float F > > cdef lambda (lambda float: float), float, float: float G > > cdef lambda (char*): lambda: (lambda float: float), float, float: > float H > > > > > > If you want a hint, the last is something that returns numerical > > integration algorithm given a string name. Yes, you could use > > typedefs, but you shouldn't have to. especially for the first. > > > Here are numba kind function annotation, I guess it may fit in here. > cdef float(float) F > cdef float(float(float), float, float) G > cdef float(float(float), float, float)(char*) H > I personally feel this kind of annotation is more packed that using ->. > > Regards, > Liu zhenhai > Here are my thoughts, for what they're worth. I actually really like the arrow syntax. It matches the way that the domains and ranges of functions are often written mathematically. On the other hand, it does conflict with the syntax from C. The syntax from Numba is a really good way to do this as well. It is just as clear and doesn't conflict with C. Strictly speaking, it isn't Python, but it would be nice to have similarities in the syntax for the different packages. The C syntax is not intuitive. The only benefit there is the overlap with C. For something this simple, learning the new syntax is easy enough. It will also The lambda syntax strikes me as an intuitive solution, but it isn't particularly clean or concise. If the syntax is going to change, it would be good to change it to something nicer than that. I don't think this one would be worth making the change. All things considered, the Numba syntax looks like the best idea. The arrows are intuitive as well, but the overlap with the syntax from C could be a bad thing. Thanks for looking into this! -Ian Henriksen > > > >> On Thu, Nov 6, 2014 at 9:59 AM, Stefan Behnel > wrote: > >> Robert Bradshaw schrieb am 06.11.2014 um 18:15: > >>>> On Thu, Nov 6, 2014 at 12:29 AM, Stefan Behnel wrote: > >>>> Robert Bradshaw schrieb am 06.11.2014 um 08:34: > >>>>> I'd like to propose a more pythonic way to declare function pointer > >>>>> types, namelye > >>>>> > >>>>> type0 (*[ident])(type1, type2, type3) > >>>>> > >>>>> would instead become > >>>>> > >>>>> (type1, type2, type3) -> type0 [ident] > >>>> > >>>> Not convinced. Looks quite magic, very different from the C that > people > >>>> would know, and takes a while reading from left to right until you > see the > >>>> "->" which is essentially the only indication of what is happening > here. > >>> > >>> I suppose it depends on whether one knows just C, is familiar with a > >>> wide variety of other (more functional) languages. I think it's a lot > >>> easier for people to read who only know Python, or just a smattering > >>> of C. > >> > >> The thing is, in order to use Cython, you currently only need to > understand > >> Python, some C/C++ and a minor bit of special Cython syntax. The more > new > >> syntax we add outside of Python/C/C++, the higher the entry bar is > raised > >> for both the "mostly Python" and "I know C" users. > > > > True. However, " -> return_annotation" is still Python syntax. > > > >>>> Also, "->" has a widely known meaning in C/C++ which makes the above > very > >>>> confusing. I think in C when thinking about function pointers, and > there is > >>>> no "def ..." keyword indication here that would switch me back to > Python > >>>> (as for signature annotations). > >>>> > >>>> I do see the problem of "where did I have to put that '*' again in > the C > >>>> declaration?", but this looks like it will produce more confusion > than it > >>>> avoids. > >>> > >>> Hmm, I hadn't thought of that meaning of ->. I don't think it would be > >>> as confusing in context, which will be immediately following cdef or > >>> in a function argument, e.g. one would write > >>> > >>> cdef (float -> float, float, float) -> float integrator = > >>> get_integrator(...) > >> > >> My brain can't even parse that. Does it join as > >> > >> (float -> (float, float, float)) > >> > >> or > >> > >> ((float -> float), float, float) > >> > >> ? > > > > The comma token remains lower precedence than about everything else. > > It's just like a parameter list > > > > cdef integrate(float -> float f, float a, float b): > > ... > > > > f is something that takes floats to floats. > > > >> A ctypedef would definitely help here. > >> > >> > >>> which doesn't to me look like member access, and is much clear than > >>> > >>> cdef float (*integrator)(float (*)(float), float, float) = > >>> get_integrator(...) > >> > >> Apart from the (*), at least this makes it clear what gets passed into > what. > >> > >> > >>> This becomes especially clear for return types, e.g. > >>> > >>> cdef ((float -> float, float, float) -> float) > >>> get_integrator(char* algorithm): > >>> if algorithm == ... > >>> > >>> vs > >>> > >>> cdef float (*get_integrator(char* algorithm))(float (*)(float), > >>> float, float): > >>> if algorithm == ... > >> > >> Same thing as above. The nesting is clearer in the latter. > > > > Clearly we think differently... I can pick apart the latter > > definition, but I have to admit it took a lot longer to type than the > > former, and I still see "cdef float blah(...., float, float): ..." > > when in fact I'm declaring something that takes a char* (or should > > that read, "takes something that dereferences to a char" :-)). > > > > I am curious, when you read "cdef int * p" do you parse this as "cdef > > (int*) p" or "cdef int (*p)" 'cause for me no matter how well I know > > it's the latter, I think the former (i.e. I think "I'm declaring a > > variable of type p that's of int pointer type.") > > > >>>> Many (or most?) function pointer type declarations will come from > some C > >>>> library's header file, so it means making it more difficult for users > to > >>>> copy over the declaration. > >>> > >>> I really hope, in the long run, people won't have to manually do these > >>> declarations. > >> > >> They'll definitely stay. At least as an advanced feature and for the > simple > >> (entry) cases, they provide both an easily understandable mapping and a > >> handy way to tune the declarations to make their usage friendlier in > Cython > >> code. > > > > I should have said they'll be less common, at least for the "copy > > header file" in which I'd rather point to the header file and list > > members to declare. For advance use (e.g. you want to mis-declare > > types) we'd still need this. > > > >>> As well as making things easier to read, part of the motivation is to > >>> come up with a syntax for declaring this type without having to add > >>> the notion of declarators (empty or not) to the grammar. Essentially > >>> everyone thinks "cdef type var" even though that's not currently the > >>> true grammar. > >> > >> True. But why not require at least a ctypedef for these things? Ad-hoc > >> declaration of C function types is just bound to get in the way of > readability. > > > > Not being able to declare this without a ctypedef would be a weakness > > in the language, especially as ctypedefs can't be declared locally. > > Also, it just defers the ugliness to elsewhere, e.g. > > > > ctypedef float (*FloatFunc)(float) > > ctypedef float (*IntegrateFunc)(FloatFunc, float, float) > > > > The reason ctypedefs help, and are so commonly used for with function > > pointers, is because the existing syntax is just so horrendously bad. > > If there's a clear way to declare a function taking a single float and > > returning a single float, no typedef needed. > > > >>> Another option that just occurred to me would be the syntax "lambda > >>> type1, type2, type3: type0" so for this example one would have > >>> > >>> cdef lambda (lambda float: float), float, float: float integrator > >>> = get_integrator(...) > >>> > >>> doesn't read as nice to me, but thought I'd throw it out there. > >> > >> It's funny that lambda uses the same expression separator as the > signature > >> annotations in Py3. Hadn't even noticed that before. > >> > >> Looks a bit funny at first sight, but definitely clearer than any of > the above. > >> > >> Stefan > >> > >> _______________________________________________ > >> cython-devel mailing list > >> cython... at python.org > >> https://mail.python.org/mailman/listinfo/cython-devel > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "cython-users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to cython-users... at googlegroups.com . > > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Fri Nov 7 09:22:05 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 7 Nov 2014 00:22:05 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <20141106213536.GA30041@pdos.lcs.mit.edu> References: <20141106213536.GA30041@pdos.lcs.mit.edu> Message-ID: On Thu, Nov 6, 2014 at 1:35 PM, C Blake wrote: > I think you should just use the C declarator syntax. Cython already > allows you to say "cdef int *foo[10]". Quick: is that a pointer to an array or 10 pointers to ints? Yes, I know what it is, but the thing is without knowing C (well) it's not immediately obvious what the precedence should be. Cython's target audience includes lots of people who don't know C well. > Declarators aren't bad - just > poorly taught, though I can see some saying those are the same thing. If they were good, they would be easy to learn, no expert teaching required. > More below. I absolutely like the declarator one the most, and the > lambda one second most. Declarator style makes it far easier to take > C code/.h files using function pointers over to Cython. So, this > discussion also depends on whether you view Cython as a bridge to > C libs or its own island/bias toward pure Py3. > > One other proposal that might appease Stefan's "Python lexical cues" > where he was missing "def" would be to take the Python3 function > definition header syntax and strip just variable names. I.e., keep the > ":"s > def foo(x: type1, y: type2, z: type3) -> type0: pass > goes to > (: type1, : type2, : type3) -> type0 [ident] > > I don't think that looks like anything else that might valid in C or > Python. It does just what C does - strip variable names from a function > header, but the ":"s maybe key your brain into a different syntax mode > since they are arguably more rare in C. (Besides stripping names, C has > some extra parens for precedence/associativity - which admittedly are > tuned to make use-expressions simpler than type-expressions.) Anyway, > I don't really like my own proposal above. Just mentioning it for > completeness in case the ":"s help anyone. > > > Robert wrote: >>I really hope, in the long run, people won't have to manually do these >>declarations. > > I agree they'll always be there somehow and also with Stefan's comments > about the entry bar. So, most people not needing them most of the time > doesn't remove the question. > > >>I am curious, when you read "cdef int * p" do you parse this as "cdef >>(int*) p" or "cdef int (*p)" 'cause for me no matter how well I know >>it's the latter, I think the former (i.e. I think "I'm declaring a >>variable of type p that's of int pointer type.") >>[..] >>Essentially everyone thinks "cdef type var" even though that's not >>currently the true grammar. >>[..] >>The reason ctypedefs help, and are so commonly used for with function >>pointers, is because the existing syntax is just so horrendously bad. >>If there's a clear way to declare a function taking a single float and >>returning a single float, no typedef needed. > > No, no, no. Look, generations of C/C++ programmers have been done > monumental disservice by textbooks/code/style guides that suggest > "int* p" is *any less* confusing than spacing "2+3/x" as "2+3 / x". > Early on in my C exposure someone pointed this out and I've never been > confused since. It's a syntax-semantics confusion. Concrete syntax > has always been right associative dereference *. In this syntax family, > the moment any operators []/*/() are introduced, you have to start > formatting it/thinking of it as a real expression, and that formatting > should track the syntax not semantics like in your head "pointer > to"/indirection speak or whatever. syntax != semantics => baddness > Spacing it as if * were left > associative to the type name is misleading at best. > > If you can only think of type decls in general as (type, var) pairs > syntactically and semantically then *of course* you find typedefs more > clear. They make pairing more explicit & shrink the expression tree to > be more trivial. You should *still* space the typedef itself in a way > suggestive of the actual concrete syntax -- "typedef int *p" (or > "ctypedef") just like you shouldn't write "2+3 / x". You should still > not essentially think of "ctypedef type var" *either*, but rather > "typedef basetype expr". In short, "essentially everyone" *should* think > and be taught and have it reinforced and "gel" by spacing that "basetype > expr" is the syntax to create a type-var bindings semantically, and only > perceive "type var" as just one simple case. You are reenforcing the point that declarators are not intuitive, but rather one has to be forcibly hit over the head with them, because they're easy to mis-understand and abuse. > Taking significance of > space in Python/Cython one step further, "int* p" could even be a hard > syntax error, but I'm not seriously proposing that change. :) Distinguishing any "a* b" from "a *b" would be a major change to the tokenizer. Indentation, not all whitespace, is significant. >I really do not think it is "essentially everyone". You know better as you > said anyway, but are in conflict with yourself, I think syntax-semantics > wise. > > Semantically, pointer indirection loads from an address before using, > and sure that can be confusing to new programmers in its own right. > Trying to unravel that confusion with anti-syntax spacing/thought > cascades the wrong way out of the whole situation and contributes > to your blocked assimilation. *If* the space guides you or barring > space parens guide you, you quickly get to never forgetting that types > are inside-out/inverse/what-I-get-if specifications. Note that this > is as it would if +,/ had somehow tricky concepts somehow "fixable" > by writing "2+3 / x" all the time. Arithmetic isn't a binding..so > the analogy is hard to complete, but my point is ad nauseum at this > stage (or even sooner! ;-). Undermine syntax with contrary whitespace > and of course it will seem bad/be harder. It might even lock you in > to thought patterns that make it really hard to think about it how > you know you "ought" to. > > Anyway, more the point, Cython has "cdef" not "py3def" or "javadef" or > whatever, after all. There is even the cdef: blocks where all the decls > and inits look eminently C-like. Having an alternative ptr syntax from C > only for functions but not arrays seems wrong to me *unless* it's really > part of a general Py3 annotations syntax move. It's about getting rid of declarators, so when one says "add types" one literally goes from def foo(arg): ... to def foo([type] arg): ... rather than def foo([base_type] [decl_stuff]arg[more_decl_stuff): ... To use your terminology, to bring the syntax in line with the semantics. I've also seen lots of people, good coders even, struggle with C function pointer declarations. > Stefan was in some > mypy-in-Py3 thread last Summer. If the situation were Cython just > moving that way to be a Py3-with-mypy compiler with some extras for C > integration, that would be a different story to me. Then trying to make > function pointers seem like definitions sans-names might make more sense. > It would really ease moves to Cython from pure Py3 users used to the type > annotations "for checking" to apply to code generation. That may be such > a big move from Cython now that it might almost deserve some "Cython3" > fork that could also drop other redundant things (like pre-memory views > NumPy integration that seems not as good as more general memviews or > something). Yeah, deleting the old way isn't going to happen any time soon. > All that being said, using lambda syntax to signify types as well as > values seems an interesting idea, if it would really work. I'm fairly convinced it would work, despite being a bit clunky looking. > But we > know C declarators will work for all things C. I don't see why we > shouldn't just stick to that. Right now and with declarators you can > tell people you just need to learn C decl syntax and Python. That's > a VERY simple thing to say (even if it requires some learning work, > and even if it isn't 100% exact). There is a lot of documentation > out there to learn C decls. Yes, the one (I'd argue only) thing C declarators has going for it is that that's what's used in C. - Robert From robertwb at gmail.com Fri Nov 7 09:24:47 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 7 Nov 2014 00:24:47 -0800 Subject: [Cython] [cython-users] Re: New function (pointer) syntax. In-Reply-To: <53004edb-16d8-4ebb-aab2-22e1248f3ec8@googlegroups.com> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> <53004edb-16d8-4ebb-aab2-22e1248f3ec8@googlegroups.com> Message-ID: On Thu, Nov 6, 2014 at 11:30 PM, Ian Henriksen wrote: > On Thursday, November 6, 2014 9:48:53 PM UTC-7, 1989lzhh wrote: >> >> > ? Nov 7, 2014?02:56?Robert Bradshaw ??? >> > >> > [Cc'ing elsewhere for more feedback. Also top-posting for initial >> > impressions before the discussion.] >> > >> > Here's some proposed function pointer syntaxes; which are the most >> > obvious to understand/read/remember? Can you figure them out? >> > >> > cdef float (*F)(float) >> > cdef float (*G)(float (*)(float), float, float) >> > cdef float ((*H)(char*))(float (*)(float), float, float) >> > >> > vs >> > >> > cdef float -> float F >> > cdef (float -> float, float, float) -> float G >> > cdef (char*) -> (float -> float, float, float) -> float H >> > >> > vs >> > >> > cdef lambda float: float F >> > cdef lambda (lambda float: float), float, float: float G >> > cdef lambda (char*): lambda: (lambda float: float), float, float: >> > float H >> > >> > >> > If you want a hint, the last is something that returns numerical >> > integration algorithm given a string name. Yes, you could use >> > typedefs, but you shouldn't have to. especially for the first. >> > >> Here are numba kind function annotation, I guess it may fit in here. >> cdef float(float) F >> cdef float(float(float), float, float) G >> cdef float(float(float), float, float)(char*) H >> I personally feel this kind of annotation is more packed that using ->. >> >> Regards, >> Liu zhenhai > > > Here are my thoughts, for what they're worth. > I actually really like the arrow syntax. It matches the way > that the domains and ranges of functions are often written > mathematically. On the other hand, it does conflict with > the syntax from C. > > The syntax from Numba is a really good way to do this as well. > It is just as clear and doesn't conflict with C. Strictly speaking, > it isn't Python, but it would be nice to have similarities in the > syntax for the different packages. > > The C syntax is not intuitive. The only benefit there is the > overlap with C. For something this simple, learning the new > syntax is easy enough. +1 > The lambda syntax strikes me as an intuitive solution, but it > isn't particularly clean or concise. If the syntax is going > to change, it would be good to change it to something nicer > than that. I don't think this one would be worth making the change. > > All things considered, the Numba syntax looks like the best idea. > The arrows are intuitive as well, but the overlap with the syntax > from C could be a bad thing. Yeah, I'm kind of leaning that way too. > Thanks for looking into this! Thanks for the feedback. - Robert From ondrej at certik.cz Fri Nov 7 18:13:54 2014 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 7 Nov 2014 10:13:54 -0700 Subject: [Cython] Cython sometimes fails to build on Travis Message-ID: Hi, Cython sometimes fails to build on Travis for various projects. If it fails, it always ends with this error: https://gist.github.com/certik/08f16dd572170c17d956 Here are some instances of this failure in other projects: https://github.com/sympy/csympy/issues/308 https://github.com/scipy/scipy/pull/4126#issuecomment-62162812 Any ideas what is going on? Thanks, Ondrej From cblake at pdos.csail.mit.edu Fri Nov 7 20:29:36 2014 From: cblake at pdos.csail.mit.edu (C Blake) Date: Fri, 7 Nov 2014 14:29:36 -0500 Subject: [Cython] New function (pointer) syntax. In-Reply-To: Message-ID: <20141107192936.GA17255@pdos.lcs.mit.edu> Robert Bradshaw robertwb at gmail.com wrote: >Quick: is that a pointer to an array or 10 pointers to ints? Yes, I >know what it is, but the thing is without knowing C (well) it's not >immediately obvious what the precedence should be. If you're gonna pick something e.g like that, it should not be something people see all the time like int main(int argc, char *argv[]). ;-) *That* I recognized correctly in far less time than it took me to read your question text. Here's a counter: printf("%s", *a[i]) - what does it do? I submit that if *a[i] is hard or not "real quick" then, eh, what the real problem here may be that you've got enough syntaxes floating in your brain that you haven't internalized all the operator rules of this one. In that case, nothing "operator-oriented" is going to make you truly happy in the long-run. I think this whole anti-declarator campaign is misguided. I agree with Greg that experienced C programmers, such as C library writers know it. In trying to "fix it", you are just squeezing complexity jello maybe to oil the squeakiest wheel of casual users. You can eek out a level or maybe two of simplicity, but you double up syntax. You can make ()s better and make []s harder..middle-scale complexity better/full scale a little worse. That kind of thing. The net result of these attempts doesn't strike me as better or worse except that different is worse. The closer to exactly C types you can get the more coherent the overall system is unless/until Python land has a commonly used type setup that is adequate. The ctypes module has a way to do this and Numba can use that, though I have little experience doing so. The context is different in that in Cython you have more opportunity to create new syntax, but should you do so? That brings up a totally other note, Could you maybe compile-time eval() ctypes stuff if you really hate C decls and want to be more pythonic? If the answer to ctypes is "Err.. not terse/compact/part of the syntax enough", well, declarators are a fine answer to terseness, that's for sure. Mostly operator text. ;-) >Cython's target audience includes lots of people who don't know C well. One leans differently based on what you think Cython "mostly" is..Bridge to libs, Its own thing, Compiled python, etc. (I know it's all the above). >If they were good, they would be easy to learn, no expert teaching required. All syntax requires learning..e.g, Stefan expressed a harder time than you visually unpacking some of the "->" exprs. All teaching can be messed up. Teaching methods can fall into bad ruts. In this case some teaching/practice actively blocks assimilation..possibly in a long-term sense like mishearing the lyrics of song or a person's name and then the actual case never sounding quite right for a long time. Or people with strong accents of your spoken past dragging you back into accented speech yourself. (Mis-)reinforced language is *tough* and can transcend good teaching. Part of this thread was you & Stefan both having an implied question of "why do I read it one way when I darn well know it's the other". I was trying to help answer that question. Maybe I'm wrong about your individual case(s). In my experience with people's trouble is that it's not just "tokens being on both sides". Most people are used to [] and () being to the right. It's active mis-reinforcement stuff like spacing/thinking const char instead of char const,.. that makes it hard. Unless you can avoid declarator style 100%, it's better to make people confront it sooner than do half-measures. > syntax != semantics => baddness It's only "misperceived/taught/assimilated semantics"-syntax divergence. I do consider the misperception unfortunate and unnecessary at the outset. The "pairing semantics" that "feel" like they diverge from the syntax for you are 'weak/vague' and *should* have lower priority in your head. It's only really 1-type, 1-var pairs in function sigs. Even you like "type varlist" in var or struct decls. I mean, c'mon: "int i, j" rocks. :) So, it's not always 1-1. Sometimes it's 1-1, sometimes 1-many aka 1 to a non-trival expr. If you go with a full expression instead of just a list, you get stuff in return as a bonus..You get stuff and lose stuff. Declarators are not some hateful, hateful thing to always avoid - there are pros and cons like anything, not "zero pros except for history" as you seem to say. Anyway, "close but different" is the order of the day and is subjective. Keeping in mind how all three function type cases look - the def/cdef, call site, type cast/type spec - should be ever present in your syntax evaluations of all you guys, and so should less trivial cases like returning a function pointer from a lookup. *Dropping/swapping* parts is arguably closer than changing how operators work. *New* operators are better than making old ones multi-personality. I don't like that cdef char *(..) approach on those grounds. -1 is my $.02. The reason the lambda approach felt interesting is because the function value cases and type cases are close in the same sort of way and Python core syntax has func values if not pointer types. This was also partly why you liked your (type1,type2)->type notation - it was just dropping py3 annotation parts. I think the biggest issue with that is py3 -> notations aren't in a whole lot of code and feel new in context rather than like just token dropping. The lambda approach at least uses a sort of new operator (new in that context). Another whole wrinkle on this/way of thinking about close but diff, if this helps anyone, is error message interpretability. If you define a method like cdef int foo(int bar, float baz) and you get back an expecting int (*)(int, float), you can compare the message and definition pretty easily. If it was (int, float) -> int, it's harder..*unless* you were mandating 'def foo(a: int, b: float)->int:'. Just another sort of angle to think at it from. The lambda spelling of that might seem kinda confusing. Cheers From njs at pobox.com Fri Nov 7 23:43:49 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 7 Nov 2014 22:43:49 +0000 Subject: [Cython] Cython sometimes fails to build on Travis In-Reply-To: References: Message-ID: On Fri, Nov 7, 2014 at 5:13 PM, Ondrej Certik wrote: > Hi, > > Cython sometimes fails to build on Travis for various projects. If it > fails, it always ends with this error: > > https://gist.github.com/certik/08f16dd572170c17d956 I don't have any insight into the error, but for CI purposes I install cython with: pip install --install-option="--no-cython-compile" cython which skips the compilation of cython itself (i.e. your cython code is still translated from .pyx to .c, but the cython compiler that does this translation runs as a pure python program). This is ridiculously faster than a normal install -- skipping compiling cython takes minutes off the build, which is far far far more than one loses by making cythonization faster. I mention it here because it seems like aside from being a good idea in general, it might also serve as a workaround for whatever this problem is. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From stefan_ml at behnel.de Sat Nov 8 00:53:38 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Nov 2014 00:53:38 +0100 Subject: [Cython] Cython sometimes fails to build on Travis In-Reply-To: References: Message-ID: <545D5B82.5050805@behnel.de> Nathaniel Smith schrieb am 07.11.2014 um 23:43: > On Fri, Nov 7, 2014 at 5:13 PM, Ondrej Certik wrote: >> Cython sometimes fails to build on Travis for various projects. If it >> fails, it always ends with this error: >> >> https://gist.github.com/certik/08f16dd572170c17d956 Might be a problem with setuptools, travis, CPython, or even Cython. > I don't have any insight into the error, but for CI purposes I install > cython with: > > pip install --install-option="--no-cython-compile" cython This is definitely the right thing to do for one-time compilations. Also, compiling the resulting C code with CFLAGS="-O0" (or "-O0 -ggdb" etc.) usually gives another speed boost for testing purposes. Stefan From robertwb at gmail.com Sat Nov 8 01:23:29 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 7 Nov 2014 16:23:29 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <20141107192936.GA17255@pdos.lcs.mit.edu> References: <20141107192936.GA17255@pdos.lcs.mit.edu> Message-ID: On Fri, Nov 7, 2014 at 11:29 AM, C Blake wrote: > Robert Bradshaw robertwb at gmail.com wrote: >>Quick: is that a pointer to an array or 10 pointers to ints? Yes, I >>know what it is, but the thing is without knowing C (well) it's not >>immediately obvious what the precedence should be. > > If you're gonna pick something e.g like that, it should not be something > people see all the time like int main(int argc, char *argv[]). ;-) > *That* I recognized correctly in far less time than it took me to read > your question text. Yeah, it (coincidentally) parses nicely as (char *) argv[], or an array of char_ptrs :-P. > Here's a counter: printf("%s", *a[i]) - what does it do? I submit that > if *a[i] is hard or not "real quick" then, eh, what the real problem > here may be that you've got enough syntaxes floating in your brain that > you haven't internalized all the operator rules of this one. In that > case, nothing "operator-oriented" is going to make you truly happy in > the long-run. > > I think this whole anti-declarator campaign is misguided. I agree with > Greg that experienced C programmers, such as C library writers know it. > In trying to "fix it", you are just squeezing complexity jello maybe to > oil the squeakiest wheel of casual users. You can eek out a level or > maybe two of simplicity, but you double up syntax. You can make ()s > better and make []s harder..middle-scale complexity better/full scale > a little worse. That kind of thing. The net result of these attempts > doesn't strike me as better or worse except that different is worse. I strongly disagree that C declarators are at even a close to an optimal point in a relatively flat parameter space. Googling "function pointer syntax" or "c declarator" reinforces the point that this is one of the most confusing and obtuse aspects of C(++) syntax. But I admit it's hard to come up with an objective measure for how good a syntax is...if it's natural to you than that's great. > The closer to exactly C types you can get the more coherent the overall > system is unless/until Python land has a commonly used type setup that > is adequate. Yes, being like C is an advantage. > The ctypes module has a way to do this and Numba can use > that, though I have little experience doing so. The context is different > in that in Cython you have more opportunity to create new syntax, but > should you do so? > > That brings up a totally other note, Could you maybe compile-time eval() > ctypes stuff if you really hate C decls and want to be more pythonic? > If the answer to ctypes is "Err.. not terse/compact/part of the syntax > enough", well, declarators are a fine answer to terseness, that's for > sure. Mostly operator text. ;-) We've considered that, primarily to support as single codebase that runs un-compiled, but if you cythonize it everything is faster (and, possibly, more typesafe). >>Cython's target audience includes lots of people who don't know C well. > > One leans differently based on what you think Cython "mostly" is..Bridge > to libs, Its own thing, Compiled python, etc. (I know it's all the above). > >>If they were good, they would be easy to learn, no expert teaching required. > > All syntax requires learning..e.g, Stefan expressed a harder time than you > visually unpacking some of the "->" exprs. All teaching can be messed up. Teacher: "So to declare a variable of type int, write "int x;" Stundet: "int x;" Teacher: "And to declare a variable of type char* write "char *s." Student: "Oh, I think I see the pattern." Teacher: "Good, let's move on..." The fact that there's such an obvious, but erroneous, pattern is (one) flaw that leads to such confusion. > Teaching methods can fall into bad ruts. In this case some teaching/practice > actively blocks assimilation..possibly in a long-term sense like mishearing > the lyrics of song or a person's name and then the actual case never sounding > quite right for a long time. Or people with strong accents of your spoken > past dragging you back into accented speech yourself. (Mis-)reinforced > language is *tough* and can transcend good teaching. Very true, but I think it's deeper than that. The problem is that we *talk* about a variable having type "char*", and the compiler *reasons* about these more complex types, but the syntax doesn't let us actually say this directly. Base types vs. complex types. Empty declarators that hold a place in the syntax tree but are not seen in the text. > Part of this thread > was you & Stefan both having an implied question of "why do I read it one > way when I darn well know it's the other". I was trying to help answer > that question. Maybe I'm wrong about your individual case(s). In my > experience with people's trouble is that it's not just "tokens being on > both sides". Most people are used to [] and () being to the right. It's > active mis-reinforcement stuff like spacing/thinking const char instead of > char const,.. that makes it hard. Unless you can avoid declarator style > 100%, it's better to make people confront it sooner than do half-measures. I'm hoping we can avoid it 100% :-) for anyone who doesn't have to actually interact with C. >> syntax != semantics => baddness > > It's only "misperceived/taught/assimilated semantics"-syntax divergence. > I do consider the misperception unfortunate and unnecessary at the outset. > The "pairing semantics" that "feel" like they diverge from the syntax for > you are 'weak/vague' and *should* have lower priority in your head. > It's only really 1-type, 1-var pairs in function sigs. Even you like > "type varlist" in var or struct decls. I mean, c'mon: "int i, j" rocks. :) Note that the "i, j" is not an expression here, the comma is a delimiter not an operator. Consider float *a = NULL which is valid but subsequent *a = NULL is not. Yep, the assignment in a declaration ignores the declarators, pretending they were attached elsewhere. > So, it's not always 1-1. Sometimes it's 1-1, sometimes 1-many aka 1 to > a non-trival expr. If you go with a full expression instead of just a > list, you get stuff in return as a bonus.. Yes, you can now declare variables of different type but the same base type on a single line... To me, "I want a set of variables of the same type" is much more common (and easy to explain) than "I want a set of variables of same basetype." > You get stuff and lose stuff. > Declarators are not some hateful, hateful thing to always avoid - there > are pros and cons like anything, not "zero pros except for history" as > you seem to say. There may be some pros, but excluding history I think they're greatly outweighed by the cons... If, hypothetically, C used "(int, int) -> int foo" syntax to declare function references, do you think anyone would have proposed "int(*foo)(int, int)" as a better alternative that we should consider? > Anyway, "close but different" is the order of the day and is subjective. > Keeping in mind how all three function type cases look - the def/cdef, > call site, type cast/type spec - should be ever present in your syntax > evaluations of all you guys, and so should less trivial cases like > returning a function pointer from a lookup. *Dropping/swapping* parts > is arguably closer than changing how operators work. *New* operators > are better than making old ones multi-personality. I don't like that > cdef char *(..) approach on those grounds. -1 is my $.02. > > The reason the lambda approach felt interesting is because the function > value cases and type cases are close in the same sort of way and Python > core syntax has func values if not pointer types. This was also partly > why you liked your (type1,type2)->type notation - it was just dropping py3 > annotation parts. I think the biggest issue with that is py3 -> notations > aren't in a whole lot of code and feel new in context rather than like > just token dropping. The lambda approach at least uses a sort of new > operator (new in that context). > > Another whole wrinkle on this/way of thinking about close but diff, if > this helps anyone, is error message interpretability. If you define a > method like cdef int foo(int bar, float baz) and you get back an expecting > int (*)(int, float), you can compare the message and definition pretty > easily. If it was (int, float) -> int, it's harder..*unless* you were > mandating 'def foo(a: int, b: float)->int:'. Just another sort of > angle to think at it from. The lambda spelling of that might seem > kinda confusing. Yes, placing the return type on the rhs is a downside of the arrow/lambda approach given what we've grown accustomed to in C (and adopted in Cython). - Robert From ondrej at certik.cz Sat Nov 8 16:29:05 2014 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 8 Nov 2014 08:29:05 -0700 Subject: [Cython] Cython sometimes fails to build on Travis In-Reply-To: References: Message-ID: Hi Nathan, On Fri, Nov 7, 2014 at 3:43 PM, Nathaniel Smith wrote: > On Fri, Nov 7, 2014 at 5:13 PM, Ondrej Certik wrote: >> Hi, >> >> Cython sometimes fails to build on Travis for various projects. If it >> fails, it always ends with this error: >> >> https://gist.github.com/certik/08f16dd572170c17d956 > > I don't have any insight into the error, but for CI purposes I install > cython with: > > pip install --install-option="--no-cython-compile" cython > > which skips the compilation of cython itself (i.e. your cython code is > still translated from .pyx to .c, but the cython compiler that does > this translation runs as a pure python program). This is ridiculously > faster than a normal install -- skipping compiling cython takes > minutes off the build, which is far far far more than one loses by > making cythonization faster. This is a great suggestion! I've implemented it in https://github.com/sympy/csympy/pull/313 and it fixes all problems (and it cut testing time in half). Of course, I think there is still some underlying bug in Cython or something else, but at least as far as Travis is concerned, this is a solution for us. Thanks, Ondrej > > I mention it here because it seems like aside from being a good idea > in general, it might also serve as a workaround for whatever this > problem is. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From yury at shurup.com Sat Nov 8 18:55:13 2014 From: yury at shurup.com (Yury V. Zaytsev) Date: Sat, 08 Nov 2014 18:55:13 +0100 Subject: [Cython] Cython sometimes fails to build on Travis In-Reply-To: <545D5B82.5050805@behnel.de> References: <545D5B82.5050805@behnel.de> Message-ID: <1415469313.2624.7.camel@newpride> On Sat, 2014-11-08 at 00:53 +0100, Stefan Behnel wrote: > > Also, compiling the resulting C code with CFLAGS="-O0" (or "-O0 -ggdb" > etc.) usually gives another speed boost for testing purposes. Just to mention it, the compilation is certainly going to be much faster, but the resulting code might end up being a lot slower, so if you then run a battery of tests that uses the module, you might end up loosing way more than you gained by cutting down the compilation time. -- Sincerely yours, Yury V. Zaytsev From stefan_ml at behnel.de Sat Nov 8 20:59:24 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Nov 2014 20:59:24 +0100 Subject: [Cython] Cython sometimes fails to build on Travis In-Reply-To: <1415469313.2624.7.camel@newpride> References: <545D5B82.5050805@behnel.de> <1415469313.2624.7.camel@newpride> Message-ID: <545E761C.6070405@behnel.de> Yury V. Zaytsev schrieb am 08.11.2014 um 18:55: > On Sat, 2014-11-08 at 00:53 +0100, Stefan Behnel wrote: >> >> Also, compiling the resulting C code with CFLAGS="-O0" (or "-O0 -ggdb" >> etc.) usually gives another speed boost for testing purposes. > > Just to mention it, the compilation is certainly going to be much > faster, but the resulting code might end up being a lot slower, so if > you then run a battery of tests that uses the module, you might end up > loosing way more than you gained by cutting down the compilation time. My assumption was that the test suite doesn't include performance tests and that it is small and concise enough to exercise most code paths only a couple of times, with only some core code paths being heavily used. That tends to be reasonable (I've yet to see a code base with a branch coverage of 100%), but will obviously not apply to all packages. If you have test for large data sets, for example, the performance may suffer noticeably, as you say. It's usually worth comparing, though. Stefan From cblake at pdos.csail.mit.edu Sun Nov 9 00:19:25 2014 From: cblake at pdos.csail.mit.edu (C Blake) Date: Sat, 8 Nov 2014 18:19:25 -0500 Subject: [Cython] New function (pointer) syntax. In-Reply-To: Message-ID: <20141108231925.GA9995@pdos.lcs.mit.edu> >But I admit it's hard to come up with an objective measure for how >good a syntax is...if it's natural to you than that's great. I think those queries you mention will mostly be biased by the squeakier wheels being more beginning people and that's not a very good argument or metric. I agree an objective measure of "goodness" or "understanding" is hard, but I happen to run Gentoo and keep my sources around. So, I did a quick grep over .c and .h files in 600 packages on my system.. pretty diverse: no one style guide or style or maintainer..Not even any very common domains..utilities, libraries, all sorts of stuff. $ grep '[a-zA-Z0-9_][a-zA-Z0-9_]\*\*[^ ]' `find -type f -name '*.[ch]'` | grep -v '/\*\*' | grep -v '\*\*/' | wc -l 3468 $ grep '[a-zA-Z0-9_][a-zA-Z0-9_] *\*\*[^ ]' `find -type f -name '*.[ch]'` | | grep -v '/\*\*' | grep -v '\*\*/' | wc -l 68900 In other words, over 95% of the instances spaced the '**' as if they knew it bound to the token on its right. ('**' is easier than '*' since the latter could be multiplies but '**' almost never is). Yes, greps are way approximate. Yes, some real parser would be better, but that just took me only a few minutes. I visually inspected what they were catching by |less instead of |wc and both cases seemed to mostly be catching decl/type uses as intended..less than a few percent error on that. If anything, the most glaring question was that 3468 "type**" cases were highly polluted with near 50% questionable "no whitespace at all" instances like (char**)a. Maintainers who know better might accept patches and lazily not fix confusing formatting. So, in a couple ways that 5% confused is an upper bound in this corpus (under a spacing = no confusion hypothesis). And, sure, confused people might format non-confusingly. And maybe '**' itself is slightly selecting for less confused people. Even so, 95..97.5% to me == "essentially no one" to you by some, let's say "not totally bonkers" measure suggests that we are just thinking of highly different populations of people. Even if you think my methods way hokey, it's probably at least suggestive "essentially no one" is a far bigger set than you thought before. So, I agree/disagree with some other things you said. Initializers are an (awfully convenient) aberration, but your example of teaching is just an example of bad teaching - so what? A list of vars of one type is easily achieved even thinking as you want with a typedef, hand having both declarators and typedefs gets you everything you want. Still, disagreements aside, I give up trying to convince you of anything beyond that you *just might* may have a very skewed perception of how confused about declarators are people coming to Cython from a C/C++ background or people who write C/C++ in general. But it seems in this arc anyway you aren't trying to target them or C code integration coherence or such...Period! As per... >I'm hoping we can avoid it 100% :-) for anyone who doesn't have to >actually interact with C. So, you're leaning hard on the Cython as a Python compiler direction. I think Cython in general should probably either be A) as C-like as possible or B) as Python-like as possible. Given your (I still think misguided) hatred of C function pointer syntax/scenario A), there's your probable answer - be as Py-like as possible. Given that, for just function types, that seems to mean either: A) the "lambda type1, type2: type3" proposal, B) what mypy does which is roughly Function[ [type1, type2], type3 ], or possibly C) what Numba does if that really catches on. or maybe the (type, type) -> rtype though that seems unpopular here, but almost surely not that "char*(..)" thing. In a few years the mypy approach may well be a PEP approved lint/typing approach and people coming from Python will at least already have maybe seen it. In dozens of emails 2..3 months ago Guido was really strongly promoting mypy, but I think it is in some kind of a-PEP-needs-to-be- written limbo. Here is a link to the relevant sub-part for those who haven't looked at it: http://www.mypy-lang.org/tutorial.html#callables I actually like A) better, but not so much better it should override what the parent to one of the two Cython syntax communities goes with. A) is really easy to describe - "just take the function value structure but use types instead of variables/expression value". There are some other styles like pytypedecl or obiwan and such that might also be worth looking into before you decide. I haven't looked at them, but thought I should mention them. From robertwb at gmail.com Sun Nov 9 03:08:06 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 8 Nov 2014 18:08:06 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: <20141108231925.GA9995@pdos.lcs.mit.edu> References: <20141108231925.GA9995@pdos.lcs.mit.edu> Message-ID: On Sat, Nov 8, 2014 at 3:19 PM, C Blake wrote: >>But I admit it's hard to come up with an objective measure for how >>good a syntax is...if it's natural to you than that's great. > > I think those queries you mention will mostly be biased by the squeakier > wheels being more beginning people and that's not a very good argument > or metric. I agree an objective measure of "goodness" or "understanding" > is hard, but I happen to run Gentoo and keep my sources around. So, I > did a quick grep over .c and .h files in 600 packages on my system.. > pretty diverse: no one style guide or style or maintainer..Not even any > very common domains..utilities, libraries, all sorts of stuff. > > $ grep '[a-zA-Z0-9_][a-zA-Z0-9_]\*\*[^ ]' `find -type f -name '*.[ch]'` | > grep -v '/\*\*' | grep -v '\*\*/' | wc -l > 3468 > > $ grep '[a-zA-Z0-9_][a-zA-Z0-9_] *\*\*[^ ]' `find -type f -name '*.[ch]'` | > | grep -v '/\*\*' | grep -v '\*\*/' | wc -l > 68900 > > In other words, over 95% of the instances spaced the '**' as if they knew > it bound to the token on its right. ('**' is easier than '*' since the > latter could be multiplies but '**' almost never is). > > Yes, greps are way approximate. Yes, some real parser would be better, > but that just took me only a few minutes. I visually inspected what they > were catching by |less instead of |wc and both cases seemed to mostly be > catching decl/type uses as intended..less than a few percent error on > that. If anything, the most glaring question was that 3468 "type**" cases > were highly polluted with near 50% questionable "no whitespace at all" > instances like (char**)a. Maintainers who know better might accept > patches and lazily not fix confusing formatting. So, in a couple ways > that 5% confused is an upper bound in this corpus (under a spacing = no > confusion hypothesis). And, sure, confused people might format > non-confusingly. And maybe '**' itself is slightly selecting for > less confused people. Yep, most C code, especially code good enough to be shipped in a standard linux distribution, is written by people who know C well. > Even so, 95..97.5% to me == "essentially no one" to you by some, let's > say "not totally bonkers" measure suggests that we are just thinking of > highly different populations of people. Even if you think my methods way > hokey, it's probably at least suggestive "essentially no one" is a far > bigger set than you thought before. So, I agree/disagree with some other > things you said. Initializers are an (awfully convenient) aberration, > but your example of teaching is just an example of bad teaching - so what? > A list of vars of one type is easily achieved even thinking as you want > with a typedef, hand having both declarators and typedefs gets you > everything you want. Still, disagreements aside, I give up trying to > convince you of anything beyond that you *just might* may have a very > skewed perception of how confused about declarators are people coming to > Cython from a C/C++ background or people who write C/C++ in general. > But it seems in this arc anyway you aren't trying to target them or > C code integration coherence or such...Period! To understand where I'm coming from, let's divide the population of Cython into three groups. (1) People who know C well. They don't have to be absolute gurus, but they're used to C declarators and can write complicated function pointers in their sleep. (2) People who know a little C. Maybe with the help of google they could parse a function pointer, but it certainly wouldn't come naturally--often all they want to do is wrap some code. (3) People who just want their Python to be fast, and don't care about C at all. Using C-style declarators primarily helps group (1), and is (IMHO) an unnecessary hurdle for group (3). Especially as passing functions around is much more common in Python. Over the years, I've seen a shift in Cython users away from (1) and towards (3). That's not to say I don't care about group (1), I do, but I also have much higher confidence that they'll be able to pick up a new syntax easily. This is not to dissimilar from supporting memory views rather than forcing users to deal with pointers directly. > As per... > >>I'm hoping we can avoid it 100% :-) for anyone who doesn't have to >>actually interact with C. > > So, you're leaning hard on the Cython as a Python compiler direction. > I think Cython in general should probably either be A) as C-like as > possible or B) as Python-like as possible. Given your (I still think > misguided) hatred of C function pointer syntax/scenario A), there's > your probable answer - be as Py-like as possible. Given that, for > just function types, that seems to mean either: > A) the "lambda type1, type2: type3" proposal, > B) what mypy does which is roughly Function[ [type1, type2], type3 ], > or possibly C) what Numba does if that really catches on. > or maybe the (type, type) -> rtype though that seems unpopular here, > but almost surely not that "char*(..)" thing. Actually type(type, type) is a type in C, just not one that can be passed by value, hence the need for function *pointers*. As function pointers are callable, and functions coerce to pointers, the distinction is not very important for the user. > In a few years the mypy approach may well be a PEP approved lint/typing > approach and people coming from Python will at least already have maybe > seen it. In dozens of emails 2..3 months ago Guido was really strongly > promoting mypy, but I think it is in some kind of a-PEP-needs-to-be- > written limbo. On that note, this is turning out to be controversial enough I should write a CEP, if no one beats me to it... https://github.com/cython/cython/wiki/enhancements > Here is a link to the relevant sub-part for those who > haven't looked at it: > > http://www.mypy-lang.org/tutorial.html#callables > > I actually like A) better, but not so much better it should override > what the parent to one of the two Cython syntax communities goes with. > A) is really easy to describe - "just take the function value structure > but use types instead of variables/expression value". > > There are some other styles like pytypedecl or obiwan and such that > might also be worth looking into before you decide. I haven't looked > at them, but thought I should mention them. Good point, I'll look at those too. - Robert From robertwb at gmail.com Mon Nov 10 18:26:32 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 10 Nov 2014 09:26:32 -0800 Subject: [Cython] [cython-users] Re: New function (pointer) syntax. In-Reply-To: References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> <545C724B.8080403@behnel.de> <926243460437170239.933899sturla.molden-gmail.com@news.gmane.org> Message-ID: On Sun, Nov 9, 2014 at 1:29 PM, Nils Bruin wrote: > On Saturday, November 8, 2014 12:15:02 PM UTC-8, Sturla Molden wrote: >> >> Presumably Cython >> needs C and C++ function and method pointer syntax for wrapping C and C++ >> libraries, so why add another one? > > > I think that's a valid point. The reality is that one of cython's strength > is that interfacing with existing C/C++ libraries is extremely > straightforward. At some point this involves translating a ".h" into a > ".pxd" file. Having many C/C++ type designations be valid in cython is > therefore a generally desirable property (especially for the inscrutable > C/C++ ones!). Thus I don't think you'd want to *deprecate* type definitions > that are presently valid in C/C++ as well as in cython. Even better would be a tool that automatically translates an .h file into a .pxd file (if the .pxd file is needed at all), pulling out all or specifically enumerated declarations. They'd only be edited by hand by experts. One could support something like from "signal.h" cimport signal rather than re-declare it. > The bar for *adding* > syntax for things that can already be expressed should be really quite high. > I'm not sure the proposal meets that bar for me. The proposal would > definitely benefit from a motivation that goes beyond "C/C++ is ugly and > hard to understand", because the reality is that cython won't be able to get > away from that entirely anyway. If you're wrapping C libraries, yes, but if you want to pass around a cdef function as a first class object as easily as you pass a standard Python callable, for people who don't already know C well. That is the use case I'm targeting, and I think the C function pointer syntax is particularly poorly suited for that. Again, I agree the "wrapping C libraries" case as a very important one we want to support, but disagree that a better syntax would be an undue burden (especially if most cases were automated away). This came up recently because I have also been toying with a formal grammar for Cython: https://github.com/robertwb/cython/blob/grammar/Cython/Parser/Grammar . Adding declarators here greatly complicates things (though feel free to try and do it cleanly). - Robert From dimpase at gmail.com Mon Nov 10 15:25:10 2014 From: dimpase at gmail.com (Dima Pasechnik) Date: Mon, 10 Nov 2014 14:25:10 +0000 (UTC) Subject: [Cython] New function (pointer) syntax. References: Message-ID: On 2014-11-06, Robert Bradshaw wrote: > I'd like to propose a more pythonic way to declare function pointer > types, namelye > > type0 (*[ident])(type1, type2, type3) > > would instead become > > (type1, type2, type3) -> type0 [ident] > > I have a pull request up at https://github.com/cython/cython/pull/333; > what do people think? we had a discussion with Volker about this few weeks ago - in my Cython code I needed to do, as he suggested, a workaround like this: int* vlamatrix "(int (*)[])" (int*) # a hack to get int (*)[] through cython This was for 2-dim arrays of variable length, and looks similar to the stuff here. IMHO it would be good to address this, too. I'd rather stick to C99 conventions in Cython. To me, using '->' for types looks way too close to what is used, with different semantics, in functional languages like Haskell or Coq. Dima From robertwb at gmail.com Mon Nov 10 21:23:37 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 10 Nov 2014 12:23:37 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: References: Message-ID: On Mon, Nov 10, 2014 at 6:25 AM, Dima Pasechnik wrote: > On 2014-11-06, Robert Bradshaw wrote: >> I'd like to propose a more pythonic way to declare function pointer >> types, namelye >> >> type0 (*[ident])(type1, type2, type3) >> >> would instead become >> >> (type1, type2, type3) -> type0 [ident] >> >> I have a pull request up at https://github.com/cython/cython/pull/333; >> what do people think? > > we had a discussion with Volker about this few weeks ago - in my Cython code I > needed to do, as he suggested, a workaround like this: > > int* vlamatrix "(int (*)[])" (int*) # a hack to get int (*)[] through cython > > This was for 2-dim arrays of variable length, and looks similar to the > stuff here. > > IMHO it would be good to address this, too. Yeah. > I'd rather stick to C99 conventions in Cython. > To me, using '->' for types looks way too close to what is used, with > different semantics, in functional languages like Haskell or Coq. Actually, the proposed use of '->' *is* identical to Haskell in semantics, i.e. "int -> int" is how you write a function taking and returning an int. Of course in Python one has a single argument tuple as input rather than currying. From rich.leigh at gmail.com Mon Nov 10 16:35:32 2014 From: rich.leigh at gmail.com (Rich Leigh) Date: Mon, 10 Nov 2014 15:35:32 +0000 Subject: [Cython] Compiler crash in cython 0.21.1 DebugTransform Message-ID: Hi all - I get the following compiler crash from cython 0.21.1 (from pip) when compiling a C++ module with debugging. It's fine without the --gdb option. [1/3] Compiling Cython CXX source for [project]... FAILED: cd [project_build] && cython --cplus -I [project_src] -2 --gdb --output-file [project_build]/[project].cxx [project_src]/[project].pyx Error compiling Cython file: ------------------------------------------------------------ ... def merge(self): cdef cpp_string ofnametmp = self.ofname + ".tmp" merge[cpp_vector[cpp_string]](self.sources, ofnametmp.c_str()) ^ ------------------------------------------------------------ [project_src]/[project].pyx:241:28: Compiler crash in DebugTransform ModuleNode.body = StatListNode([project].pyx:1:0) StatListNode.stats[12] = StatListNode([project].pyx:227:5) StatListNode.stats[0] = CompilerDirectivesNode([project].pyx:227:5) CompilerDirectivesNode.body = StatListNode([project].pyx:227:5) StatListNode.stats[0] = CClassDefNode([project].pyx:227:5, as_name = u'Merge', class_name = u'Merge', decorators = [...]/0, module_name = u'', visibility = u'private') CClassDefNode.body = StatListNode([project].pyx:228:4) StatListNode.stats[1] = DefNode([project].pyx:238:4, modifiers = [...]/0, module_name = u'[Project]', name = u'merge', num_required_args = 1, py_wrapper_required = True, qualname = u'Merge.merge', reqd_kw_flags_cname = '0', used = True) DefNode.body = StatListNode([project].pyx:239:8) StatListNode.stats[0] = ExprStatNode([project].pyx:240:41) ExprStatNode.expr = SimpleCallNode([project].pyx:240:41, analysed = True, is_temp = 1, use_managed_ref = True) SimpleCallNode.function = IndexNode([project].pyx:240:17, is_called = 1, is_subscript = True, result_is_used = True, type_indices = [...]/1, use_managed_ref = True) IndexNode.index = IndexNode([project].pyx:240:28, is_subscript = True, result_is_used = True, use_managed_ref = True) IndexNode.base = NameNode([project].pyx:240:28, cf_maybe_null = True, is_name = True, name = u'cpp_vector', result_is_used = True, use_managed_ref = True) Compiler crash traceback from this point on: File "Cython/Compiler/Visitor.py", line 173, in Cython.Compiler.Visitor.TreeVisitor._visit (cython/Cython/Compiler/Visitor.c:4449) return handler_method(obj) File "python2.7/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 2937, in visit_NameNode node.type.is_cfunction and AttributeError: 'NoneType' object has no attribute 'is_cfunction' In my pyx file, cpp_vector and cpp_string are as follows: from libcpp.vector cimport vector as cpp_vector from libcpp.string cimport string as cpp_string and merge (my C++ function, though I've also got the python def above of a method with the same name) has this signature: cdef void merge[T](const T&, const char *) except+ I've managed to fix it by turning line 2937 in ParseTreeTransforms.py from 2937: node.type.is_cfunction and to 2937: getattr(node.type, 'is_cfunction', False) and which seems to do the trick. I did a quick google to see if anybody else had this issue, but it seems like a fresh bug. Not sure if this is the correct solution though, or whether I've simply buried an underlying issue. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dimpase+github at gmail.com Mon Nov 10 22:10:55 2014 From: dimpase+github at gmail.com (Dima Pasechnik) Date: Mon, 10 Nov 2014 21:10:55 +0000 Subject: [Cython] New function (pointer) syntax. In-Reply-To: References: Message-ID: On 10 November 2014 20:23, Robert Bradshaw wrote: > On Mon, Nov 10, 2014 at 6:25 AM, Dima Pasechnik wrote: >> On 2014-11-06, Robert Bradshaw wrote: >>> I'd like to propose a more pythonic way to declare function pointer >>> types, namelye >>> >>> type0 (*[ident])(type1, type2, type3) >>> >>> would instead become >>> >>> (type1, type2, type3) -> type0 [ident] >>> >>> I have a pull request up at https://github.com/cython/cython/pull/333; >>> what do people think? >> >> we had a discussion with Volker about this few weeks ago - in my Cython code I >> needed to do, as he suggested, a workaround like this: >> >> int* vlamatrix "(int (*)[])" (int*) # a hack to get int (*)[] through cython >> >> This was for 2-dim arrays of variable length, and looks similar to the >> stuff here. >> >> IMHO it would be good to address this, too. > > Yeah. > >> I'd rather stick to C99 conventions in Cython. >> To me, using '->' for types looks way too close to what is used, with >> different semantics, in functional languages like Haskell or Coq. > > Actually, the proposed use of '->' *is* identical to Haskell in > semantics, i.e. "int -> int" is how you write a function taking and > returning an int. Of course in Python one has a single argument tuple > as input rather than currying. well, I don't really see why ident is put at the end of the statement. As you know, Haskell-like it would be ident :: (type1, type2, type3) -> type0 if you work in postscript, then indeed putting ident at the end would be natural :-) Dima From robertwb at gmail.com Tue Nov 11 08:20:08 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 10 Nov 2014 23:20:08 -0800 Subject: [Cython] New function (pointer) syntax. In-Reply-To: References: Message-ID: On Mon, Nov 10, 2014 at 1:10 PM, Dima Pasechnik wrote: > On 10 November 2014 20:23, Robert Bradshaw wrote: >> On Mon, Nov 10, 2014 at 6:25 AM, Dima Pasechnik wrote: >>> On 2014-11-06, Robert Bradshaw wrote: >>>> I'd like to propose a more pythonic way to declare function pointer >>>> types, namelye >>>> >>>> type0 (*[ident])(type1, type2, type3) >>>> >>>> would instead become >>>> >>>> (type1, type2, type3) -> type0 [ident] >>>> >>>> I have a pull request up at https://github.com/cython/cython/pull/333; >>>> what do people think? >>> >>> we had a discussion with Volker about this few weeks ago - in my Cython code I >>> needed to do, as he suggested, a workaround like this: >>> >>> int* vlamatrix "(int (*)[])" (int*) # a hack to get int (*)[] through cython >>> >>> This was for 2-dim arrays of variable length, and looks similar to the >>> stuff here. >>> >>> IMHO it would be good to address this, too. >> >> Yeah. >> >>> I'd rather stick to C99 conventions in Cython. >>> To me, using '->' for types looks way too close to what is used, with >>> different semantics, in functional languages like Haskell or Coq. >> >> Actually, the proposed use of '->' *is* identical to Haskell in >> semantics, i.e. "int -> int" is how you write a function taking and >> returning an int. Of course in Python one has a single argument tuple >> as input rather than currying. > > well, I don't really see why ident is put at the end of the statement. > As you know, Haskell-like it would be > > ident :: (type1, type2, type3) -> type0 > > if you work in postscript, then indeed putting ident at the end > would be natural :-) Ident is put at the end because changing that would be much, much more invasive, e.g. "cdef a, b, c [token] int", though there are definitely advantages of doing it that way. From daniele at grinta.net Tue Nov 11 18:01:34 2014 From: daniele at grinta.net (Daniele Nicolodi) Date: Tue, 11 Nov 2014 18:01:34 +0100 Subject: [Cython] [cython-users] Re: New function (pointer) syntax. In-Reply-To: <54623F6E.3000302@grinta.net> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> <53004edb-16d8-4ebb-aab2-22e1248f3ec8@googlegroups.com> <54623F6E.3000302@grinta.net> Message-ID: <546240EE.7070907@grinta.net> On 11/11/14 17:55, Daniele Nicolodi wrote: > On 07/11/14 08:30, Ian Henriksen wrote: >> The syntax from Numba is a really good way to do this as well. >> It is just as clear and doesn't conflict with C. Strictly speaking, >> it isn't Python, but it would be nice to have similarities in the >> syntax for the different packages. > > What would be the Numbe syntax? I had a quick look at Numba > documentation but I didn't find where this is described in detail. Nevermind, it is so simple that I overlooked it in the documentation. Cheers, Daniele From daniele at grinta.net Tue Nov 11 17:55:10 2014 From: daniele at grinta.net (Daniele Nicolodi) Date: Tue, 11 Nov 2014 17:55:10 +0100 Subject: [Cython] [cython-users] Re: New function (pointer) syntax. In-Reply-To: <53004edb-16d8-4ebb-aab2-22e1248f3ec8@googlegroups.com> References: <545B3179.7010802@behnel.de> <545BB70A.5030900@behnel.de> <3DF00661-18ED-4AE6-B6A7-68E00465D840@gmail.com> <53004edb-16d8-4ebb-aab2-22e1248f3ec8@googlegroups.com> Message-ID: <54623F6E.3000302@grinta.net> On 07/11/14 08:30, Ian Henriksen wrote: > The syntax from Numba is a really good way to do this as well. > It is just as clear and doesn't conflict with C. Strictly speaking, > it isn't Python, but it would be nice to have similarities in the > syntax for the different packages. What would be the Numbe syntax? I had a quick look at Numba documentation but I didn't find where this is described in detail. Cheers, Daniele From matthew.brett at gmail.com Fri Nov 14 22:38:49 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 14 Nov 2014 13:38:49 -0800 Subject: [Cython] Cython sometimes fails to build on Travis In-Reply-To: <545E761C.6070405@behnel.de> References: <545D5B82.5050805@behnel.de> <1415469313.2624.7.camel@newpride> <545E761C.6070405@behnel.de> Message-ID: Hi, On Sat, Nov 8, 2014 at 11:59 AM, Stefan Behnel wrote: > Yury V. Zaytsev schrieb am 08.11.2014 um 18:55: >> On Sat, 2014-11-08 at 00:53 +0100, Stefan Behnel wrote: >>> >>> Also, compiling the resulting C code with CFLAGS="-O0" (or "-O0 -ggdb" >>> etc.) usually gives another speed boost for testing purposes. >> >> Just to mention it, the compilation is certainly going to be much >> faster, but the resulting code might end up being a lot slower, so if >> you then run a battery of tests that uses the module, you might end up >> loosing way more than you gained by cutting down the compilation time. > > My assumption was that the test suite doesn't include performance tests and > that it is small and concise enough to exercise most code paths only a > couple of times, with only some core code paths being heavily used. That > tends to be reasonable (I've yet to see a code base with a branch coverage > of 100%), but will obviously not apply to all packages. If you have test > for large data sets, for example, the performance may suffer noticeably, as > you say. > > It's usually worth comparing, though. I have the same build error on a real machine: http://nipy.bic.berkeley.edu/builders/dipy-py3.4/builds/49/steps/shell_5/logs/stdio Is there anything you can suggest to help debug? Cheers, Matthew From fnc4 at cornell.edu Fri Nov 14 13:46:33 2014 From: fnc4 at cornell.edu (Favian Contreras) Date: Fri, 14 Nov 2014 04:46:33 -0800 Subject: [Cython] Location where extern declarations are parsed Message-ID: Hello all, Does anyone know where the (c++) external declarations are parsed? I am specifically looking for where external function pointers are parsed by the compiler. I noticed that all functions are parsed in some way in Symtab.py, but this includes all of the inherited methods from python and the c files within Cython, and I'm thinking there is a more specific place where external function pointers parsed. Cheers, Favian Contreras -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Sat Nov 15 17:48:44 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 15 Nov 2014 08:48:44 -0800 Subject: [Cython] Location where extern declarations are parsed In-Reply-To: References: Message-ID: On Fri, Nov 14, 2014 at 4:46 AM, Favian Contreras wrote: > Hello all, > > Does anyone know where the (c++) external declarations are parsed? I am > specifically looking for where external function pointers are parsed by the > compiler. I noticed that all functions are parsed in some way in Symtab.py, > but this includes all of the inherited methods from python and the c files > within Cython, and I'm thinking there is a more specific place where > external function pointers parsed. Look in Cython/Compiler/Parsing.py. https://github.com/cython/cython/blob/04e462aa939e0d4639dd4feac5b7e6aeff8399ad/Cython/Compiler/Parsing.py#L2825 During the analyse_declarations phase they're placed in the module's scope in Symtab. From michael at ensslin.cc Sun Nov 30 16:14:08 2014 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Sun, 30 Nov 2014 16:14:08 +0100 Subject: [Cython] cython --cplus --embed generates invalid code Message-ID: <547B3440.9040903@ensslin.cc> Hi, on my system, --embed does not work with --cplus. How to reproduce: Any valid pyx file works: $ rm -f test.pyx; touch test.pyx $ cython --embed --cplus test.pyx $ g++ -c test.cpp -I/usr/include/python3.4m test.cpp: In function ?wchar_t* __Pyx_char2wchar(char*)?: test.cpp:945:41: error: invalid conversion from ?void*? to ?wchar_t*? [-fpermissive] res = malloc(argsize*sizeof(wchar_t)); $ clang++ -c test.cpp -I/usr/include/python3.4m test.cpp:945:9: error: assigning to 'wchar_t *' from incompatible type 'void *' res = malloc(argsize*sizeof(wchar_t)); The issue can easily be fixed by manually casting the malloc result to (wchar_t *). In C it is not recommended to cast the malloc result, while in C++ it is required (and malloc is discouraged). System info: $ cython --version Cython version 0.21.1 $ g++ --version g++ (Debian 4.9.2-2) 4.9.2 $ python3 --version Python 3.4.2 $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux unstable (sid) Release: unstable Codename: sid ~ Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: