From erik.m.bray at gmail.com Tue May 3 06:04:56 2016 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 3 May 2016 12:04:56 +0200 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: References:

Message-ID: On Thu, Apr 28, 2016 at 9:29 AM, Robert Bradshaw wrote: > On Wed, Apr 27, 2016 at 3:07 AM, Erik Bray wrote: >> >> On Tue, Apr 26, 2016 at 10:55 PM, Robert Bradshaw >> wrote: >> > On Tue, Apr 26, 2016 at 8:36 AM, Erik Bray >> > wrote: >> >> >> >> On Tue, Apr 26, 2016 at 5:16 PM, Dima Pasechnik >> >> wrote: >> >> > Hi, >> >> > certainly we did something with Sage on cygwin to work around >> >> > these... >> >> > Just in case, >> >> >> >> From what I can tell there are several places where Sage has hacked >> >> around this issue in different packages, but it's not doing anything >> >> specifically with it for Cython. Sage uses the Cephes math lib to >> >> support these functions on FreeBSD--and apparently used to use it on >> >> Cygwin too, but disabled that for some reason. >> >> >> >> Regardless, Cython should ultimately do something sensible in these >> >> cases. >> >> >> >> My general thinking is that in cases where Cython generates code >> >> containing C math functions, it ought to support a fallback. This >> >> will require some feature checks so that Cython can generate wrappers, >> >> when necessary, around the double versions of those functions (as >> >> Numpy currently does). >> > >> > >> > Let's make things concrete. You're complaining that something like >> > >> > cdef extern from "math.h": >> > long double sqrtl(long double) >> > >> > def foo(long double x): >> > return sqrtl(x) >> > >> > Doesn't work on Cygwin? >> > >> > The same is true for *any* C function that you use that's not totally >> > portable (this is the bane of trying to use C). I don't think Cython >> > should >> > be detecting this and substituting a (less accurate) sqrt for sqrtl in >> > this >> > case. If you want do do this, write your own headers that >> > (conditionally) >> > define these things however you want. >> >> No, not at all. That would be silly. Let me be clearer... >> >> > Or, are you complaining that Cython's test suite doesn't pass on some >> > Cygwin >> > because there are tests of features not available on Cygwin? (Your >> > original >> > email isn't clear.) If so, the test framework can be set up to exclude >> > these >> > tests on that platform. >> >> There are really two concerns. This is one of them, yes. The first >> question is what would be the best way to exclude certain tests on >> certain platforms? There's no clear documentation on that (which is >> fine, but it's why I'm asking :) > > > Probably add a tag and then modify runtests.py to detect Cygwin and > automatically add this to the exclusion list. Okay. >> The second concern, and more serious, is that there *are* cases where >> Cython generates code that uses long double functions where, for >> example, long double is passed as an argument. For example the >> following code >> >> def truncate_long_double(long double x): >> cdef float r = int(x) >> return r >> >> compiles to something that includes: >> >> /* "truncl.pyx":2 >> * def truncate_long_double(long double x): >> * cdef float r = int(x) # <<<<<<<<<<<<<< >> * return r >> */ >> __pyx_t_1 = truncl(__pyx_v_x); >> __pyx_v_r = __pyx_t_1; >> >> /* "truncl.pyx":3 >> * def truncate_long_double(long double x): >> * cdef float r = int(x) >> * return r # <<<<<<<<<<<<<< >> */ >> __Pyx_XDECREF(__pyx_r); >> __pyx_t_2 = PyFloat_FromDouble(__pyx_v_r); if >> (unlikely(!__pyx_t_2)) __PYX_ERR(0, 3, __pyx_L1_error) >> >> It's not not clear what the best way would be to *not* use this code >> on platforms where truncl missing (and there are a handful of other >> examples besides truncl). This code is generated by an optimization >> that was added here: http://trac.cython.org/ticket/400 In this case >> replacing truncl with trunc shouldn't hurt. > > > Here, yes, but in general truncl(x) != trunc(x) for, say, 2**54 + 1. > >> It's not uncommon (I think) to ship pre-cythonized C sources in a >> package's source distribution so that it can be installed (especially >> by pip) without requiring the user to have Cython. This code will >> fail to compile on platforms like Cygwin. >> >> But it's not clear how to handle this at Cythonization time either >> since it doesn't know in advance what platform it will be targeting. >> I'm suggesting that maybe rather than using these functions directly >> Cython should generate some fallbacks for where they're not available, >> or at least have the option to. > > > The safest is to never use the long double type in this case--we assume that > if long double is present (used), so are the corresponding functions in > math.h (which is wrong for this platform, but if we need to use powl I don't > know what else to do). Alternatively, ship your own (conditionally defined) > fallbacks. I agree that it would be safest not to use long double where long double functions are not support, and an application that does should probably check for that. The problem is that Cython is generating code that simply *cannot* be compiled on such platforms. My suggestion would be to replace the current calls to [trunc,fmod,pow]l to the appropriate __Pyx_%(func)_%(type) wrapper which would provide two alternatives: One that works properly, and one that falls back to using the double version of the function (Numpy does this so there is some precedent). I would go further and suggest throwing up a pre-compiler warning when using the double versions. The only question is what condition (in my mind) is what to use as a condition for selecting the fallbacks. For now I'm thinking something quite specific like #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) since I'm apparently the only person who cares about this at the moment, and Cygwin is the platform I'm targeting. Again, my only real concern right now is that Cython generates code that can be compiled on Cygwin. Best, Erik From robertwb at gmail.com Tue May 3 13:15:43 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 3 May 2016 10:15:43 -0700 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: References:

Message-ID: On Tue, May 3, 2016 at 3:04 AM, Erik Bray wrote: > On Thu, Apr 28, 2016 at 9:29 AM, Robert Bradshaw > wrote: > > On Wed, Apr 27, 2016 at 3:07 AM, Erik Bray > wrote: > >> > >> On Tue, Apr 26, 2016 at 10:55 PM, Robert Bradshaw > >> wrote: > >> > On Tue, Apr 26, 2016 at 8:36 AM, Erik Bray > >> > wrote: > >> >> > >> >> On Tue, Apr 26, 2016 at 5:16 PM, Dima Pasechnik > >> >> wrote: > >> >> > Hi, > >> >> > certainly we did something with Sage on cygwin to work around > >> >> > these... > >> >> > Just in case, > >> >> > >> >> From what I can tell there are several places where Sage has hacked > >> >> around this issue in different packages, but it's not doing anything > >> >> specifically with it for Cython. Sage uses the Cephes math lib to > >> >> support these functions on FreeBSD--and apparently used to use it on > >> >> Cygwin too, but disabled that for some reason. > >> >> > >> >> Regardless, Cython should ultimately do something sensible in these > >> >> cases. > >> >> > >> >> My general thinking is that in cases where Cython generates code > >> >> containing C math functions, it ought to support a fallback. This > >> >> will require some feature checks so that Cython can generate > wrappers, > >> >> when necessary, around the double versions of those functions (as > >> >> Numpy currently does). > >> > > >> > > >> > Let's make things concrete. You're complaining that something like > >> > > >> > cdef extern from "math.h": > >> > long double sqrtl(long double) > >> > > >> > def foo(long double x): > >> > return sqrtl(x) > >> > > >> > Doesn't work on Cygwin? > >> > > >> > The same is true for *any* C function that you use that's not totally > >> > portable (this is the bane of trying to use C). I don't think Cython > >> > should > >> > be detecting this and substituting a (less accurate) sqrt for sqrtl in > >> > this > >> > case. If you want do do this, write your own headers that > >> > (conditionally) > >> > define these things however you want. > >> > >> No, not at all. That would be silly. Let me be clearer... > >> > >> > Or, are you complaining that Cython's test suite doesn't pass on some > >> > Cygwin > >> > because there are tests of features not available on Cygwin? (Your > >> > original > >> > email isn't clear.) If so, the test framework can be set up to exclude > >> > these > >> > tests on that platform. > >> > >> There are really two concerns. This is one of them, yes. The first > >> question is what would be the best way to exclude certain tests on > >> certain platforms? There's no clear documentation on that (which is > >> fine, but it's why I'm asking :) > > > > > > Probably add a tag and then modify runtests.py to detect Cygwin and > > automatically add this to the exclusion list. > > Okay. > > >> The second concern, and more serious, is that there *are* cases where > >> Cython generates code that uses long double functions where, for > >> example, long double is passed as an argument. For example the > >> following code > >> > >> def truncate_long_double(long double x): > >> cdef float r = int(x) > >> return r > >> > >> compiles to something that includes: > >> > >> /* "truncl.pyx":2 > >> * def truncate_long_double(long double x): > >> * cdef float r = int(x) # <<<<<<<<<<<<<< > >> * return r > >> */ > >> __pyx_t_1 = truncl(__pyx_v_x); > >> __pyx_v_r = __pyx_t_1; > >> > >> /* "truncl.pyx":3 > >> * def truncate_long_double(long double x): > >> * cdef float r = int(x) > >> * return r # <<<<<<<<<<<<<< > >> */ > >> __Pyx_XDECREF(__pyx_r); > >> __pyx_t_2 = PyFloat_FromDouble(__pyx_v_r); if > >> (unlikely(!__pyx_t_2)) __PYX_ERR(0, 3, __pyx_L1_error) > >> > >> It's not not clear what the best way would be to *not* use this code > >> on platforms where truncl missing (and there are a handful of other > >> examples besides truncl). This code is generated by an optimization > >> that was added here: http://trac.cython.org/ticket/400 In this case > >> replacing truncl with trunc shouldn't hurt. > > > > > > Here, yes, but in general truncl(x) != trunc(x) for, say, 2**54 + 1. > > > >> It's not uncommon (I think) to ship pre-cythonized C sources in a > >> package's source distribution so that it can be installed (especially > >> by pip) without requiring the user to have Cython. This code will > >> fail to compile on platforms like Cygwin. > >> > >> But it's not clear how to handle this at Cythonization time either > >> since it doesn't know in advance what platform it will be targeting. > >> I'm suggesting that maybe rather than using these functions directly > >> Cython should generate some fallbacks for where they're not available, > >> or at least have the option to. > > > > > > The safest is to never use the long double type in this case--we assume > that > > if long double is present (used), so are the corresponding functions in > > math.h (which is wrong for this platform, but if we need to use powl I > don't > > know what else to do). Alternatively, ship your own (conditionally > defined) > > fallbacks. > > I agree that it would be safest not to use long double where long > double functions are not support, and an application that does should > probably check for that. > > The problem is that Cython is generating code that simply *cannot* be > compiled on such platforms. Does Cython ever generate such code when long doubles are not used in the source? If so, we should fix that. But I don't think Cython should be swapping in double versions for long double operations. > My suggestion would be to replace the > current calls to [trunc,fmod,pow]l to the appropriate > __Pyx_%(func)_%(type) wrapper which would provide two alternatives: > One that works properly, and one that falls back to using the double > version of the function (Numpy does this so there is some precedent). > I would go further and suggest throwing up a pre-compiler warning when > using the double versions. > > The only question is what condition (in my mind) is what to use as a > condition for selecting the fallbacks. For now I'm thinking something > quite specific like > > #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) > > since I'm apparently the only person who cares about this at the > moment, and Cygwin is the platform I'm targeting. > > Again, my only real concern right now is that Cython generates code > that can be compiled on Cygwin. > > If you want this behavior, wouldn't including a header #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) #define truncl trunc #define fmodl fmod #define powl pow #endif as part of your package do the job? -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed May 4 11:01:03 2016 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 4 May 2016 17:01:03 +0200 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: References:

Message-ID: On Tue, May 3, 2016 at 7:15 PM, Robert Bradshaw wrote: > On Tue, May 3, 2016 at 3:04 AM, Erik Bray wrote: >> >> On Thu, Apr 28, 2016 at 9:29 AM, Robert Bradshaw >> wrote: >> > On Wed, Apr 27, 2016 at 3:07 AM, Erik Bray >> > wrote: >> >> >> >> On Tue, Apr 26, 2016 at 10:55 PM, Robert Bradshaw >> >> wrote: >> >> > On Tue, Apr 26, 2016 at 8:36 AM, Erik Bray >> >> > wrote: >> >> >> >> >> >> On Tue, Apr 26, 2016 at 5:16 PM, Dima Pasechnik >> >> >> wrote: >> >> >> > Hi, >> >> >> > certainly we did something with Sage on cygwin to work around >> >> >> > these... >> >> >> > Just in case, >> >> >> >> >> >> From what I can tell there are several places where Sage has hacked >> >> >> around this issue in different packages, but it's not doing anything >> >> >> specifically with it for Cython. Sage uses the Cephes math lib to >> >> >> support these functions on FreeBSD--and apparently used to use it on >> >> >> Cygwin too, but disabled that for some reason. >> >> >> >> >> >> Regardless, Cython should ultimately do something sensible in these >> >> >> cases. >> >> >> >> >> >> My general thinking is that in cases where Cython generates code >> >> >> containing C math functions, it ought to support a fallback. This >> >> >> will require some feature checks so that Cython can generate >> >> >> wrappers, >> >> >> when necessary, around the double versions of those functions (as >> >> >> Numpy currently does). >> >> > >> >> > >> >> > Let's make things concrete. You're complaining that something like >> >> > >> >> > cdef extern from "math.h": >> >> > long double sqrtl(long double) >> >> > >> >> > def foo(long double x): >> >> > return sqrtl(x) >> >> > >> >> > Doesn't work on Cygwin? >> >> > >> >> > The same is true for *any* C function that you use that's not totally >> >> > portable (this is the bane of trying to use C). I don't think Cython >> >> > should >> >> > be detecting this and substituting a (less accurate) sqrt for sqrtl >> >> > in >> >> > this >> >> > case. If you want do do this, write your own headers that >> >> > (conditionally) >> >> > define these things however you want. >> >> >> >> No, not at all. That would be silly. Let me be clearer... >> >> >> >> > Or, are you complaining that Cython's test suite doesn't pass on some >> >> > Cygwin >> >> > because there are tests of features not available on Cygwin? (Your >> >> > original >> >> > email isn't clear.) If so, the test framework can be set up to >> >> > exclude >> >> > these >> >> > tests on that platform. >> >> >> >> There are really two concerns. This is one of them, yes. The first >> >> question is what would be the best way to exclude certain tests on >> >> certain platforms? There's no clear documentation on that (which is >> >> fine, but it's why I'm asking :) >> > >> > >> > Probably add a tag and then modify runtests.py to detect Cygwin and >> > automatically add this to the exclusion list. >> >> Okay. >> >> >> The second concern, and more serious, is that there *are* cases where >> >> Cython generates code that uses long double functions where, for >> >> example, long double is passed as an argument. For example the >> >> following code >> >> >> >> def truncate_long_double(long double x): >> >> cdef float r = int(x) >> >> return r >> >> >> >> compiles to something that includes: >> >> >> >> /* "truncl.pyx":2 >> >> * def truncate_long_double(long double x): >> >> * cdef float r = int(x) # <<<<<<<<<<<<<< >> >> * return r >> >> */ >> >> __pyx_t_1 = truncl(__pyx_v_x); >> >> __pyx_v_r = __pyx_t_1; >> >> >> >> /* "truncl.pyx":3 >> >> * def truncate_long_double(long double x): >> >> * cdef float r = int(x) >> >> * return r # <<<<<<<<<<<<<< >> >> */ >> >> __Pyx_XDECREF(__pyx_r); >> >> __pyx_t_2 = PyFloat_FromDouble(__pyx_v_r); if >> >> (unlikely(!__pyx_t_2)) __PYX_ERR(0, 3, __pyx_L1_error) >> >> >> >> It's not not clear what the best way would be to *not* use this code >> >> on platforms where truncl missing (and there are a handful of other >> >> examples besides truncl). This code is generated by an optimization >> >> that was added here: http://trac.cython.org/ticket/400 In this case >> >> replacing truncl with trunc shouldn't hurt. >> > >> > >> > Here, yes, but in general truncl(x) != trunc(x) for, say, 2**54 + 1. >> > >> >> It's not uncommon (I think) to ship pre-cythonized C sources in a >> >> package's source distribution so that it can be installed (especially >> >> by pip) without requiring the user to have Cython. This code will >> >> fail to compile on platforms like Cygwin. >> >> >> >> But it's not clear how to handle this at Cythonization time either >> >> since it doesn't know in advance what platform it will be targeting. >> >> I'm suggesting that maybe rather than using these functions directly >> >> Cython should generate some fallbacks for where they're not available, >> >> or at least have the option to. >> > >> > >> > The safest is to never use the long double type in this case--we assume >> > that >> > if long double is present (used), so are the corresponding functions in >> > math.h (which is wrong for this platform, but if we need to use powl I >> > don't >> > know what else to do). Alternatively, ship your own (conditionally >> > defined) >> > fallbacks. >> >> I agree that it would be safest not to use long double where long >> double functions are not support, and an application that does should >> probably check for that. >> >> The problem is that Cython is generating code that simply *cannot* be >> compiled on such platforms. > > > Does Cython ever generate such code when long doubles are not used in the > source? If so, we should fix that. But I don't think Cython should be > swapping in double versions for long double operations. No, but the problem is that Cython is a pretty heavy layer of abstraction. In the example I gave no long double function was ever used explicitly--it was added by a compiler optimization. I probably wouldn't write that code thinking that I will need a workaround on platforms that don't have truncl. In the slightly more explicit cases of fmod and pow I might agree. Though there still isn't a great way to anticipate the need to write separate handling for long double functions into a module, for a platform that doesn't support them. I can't write something in a Cython module like: if sys.platform != 'cygwin': def my_long_double_func(long double x): .... Maybe would be nice to have a simple compiler directive that outright disables compiling a function depending on a platform check. (I think this would also be useful for the test framework--the current system of module-level tags is not fine-grained as I would like, I'm finding. >> My suggestion would be to replace the >> current calls to [trunc,fmod,pow]l to the appropriate >> __Pyx_%(func)_%(type) wrapper which would provide two alternatives: >> One that works properly, and one that falls back to using the double >> version of the function (Numpy does this so there is some precedent). >> I would go further and suggest throwing up a pre-compiler warning when >> using the double versions. >> >> The only question is what condition (in my mind) is what to use as a >> condition for selecting the fallbacks. For now I'm thinking something >> quite specific like >> >> #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) >> >> since I'm apparently the only person who cares about this at the >> moment, and Cygwin is the platform I'm targeting. >> >> Again, my only real concern right now is that Cython generates code >> that can be compiled on Cygwin. > > > If you want this behavior, wouldn't including a header > > #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) > #define truncl trunc > #define fmodl fmod > #define powl pow > #endif > > as part of your package do the job? That's a reasonable workaround. But I would argue that should be baked into Cythonized code, at least optionally, and should generate a warning. Best, Erik From christoph at grothesque.org Tue May 3 10:19:14 2016 From: christoph at grothesque.org (Christoph Groth) Date: Tue, 03 May 2016 16:19:14 +0200 Subject: [Cython] missing cimport in module '.' - bug? Message-ID: <87lh3rw7fh.fsf@grothesque.org> Hello, The setup.py script of our package "kwant" uses Cython.Build.cythonize. Everything works well, except that for pyx files that cimport a module from "." an error message is generated. E.g. for the file "kwant/linalg/lapack.pyx" that contains the line from . cimport f_lapack The following message appears whenever setup.py is run: missing cimport in module '.': kwant/linalg/lapack.pyx No error message appears for files that cimport only from somewhere else than ".", e.g.: from .defs cimport gint The said messages do not seem to cause any problems, but they are of course somewhat of a nuiscance. They seem to be created by Cython.Build.Dependencies.cimported_files(). The culprit seems to be the function find_pxd in that module that does not seem to handle properly the case when module is ".". To check out the problem, download "kwant" from PyPI and execute, for example, "./setup.py --help". Cheers, Christoph From robertwb at math.washington.edu Tue May 10 02:01:36 2016 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 May 2016 23:01:36 -0700 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: References:

Message-ID: On Wed, May 4, 2016 at 8:01 AM, Erik Bray wrote: > On Tue, May 3, 2016 at 7:15 PM, Robert Bradshaw wrote: >> On Tue, May 3, 2016 at 3:04 AM, Erik Bray wrote: >>> I agree that it would be safest not to use long double where long >>> double functions are not support, and an application that does should >>> probably check for that. >>> >>> The problem is that Cython is generating code that simply *cannot* be >>> compiled on such platforms. >> >> Does Cython ever generate such code when long doubles are not used in the >> source? If so, we should fix that. But I don't think Cython should be >> swapping in double versions for long double operations. > > No, but the problem is that Cython is a pretty heavy layer of > abstraction. In the example I gave no long double function was ever > used explicitly--it was added by a compiler optimization. I probably > wouldn't write that code thinking that I will need a workaround on > platforms that don't have truncl. I would argue that if you don't have truncl, you shouldn't be using long double. Possibly this particular use of truncl could be worked around (though note that using trunc is not safe) but it's starting to get on shaky ground. > In the slightly more explicit cases of fmod and pow I might agree. > Though there still isn't a great way to anticipate the need to write > separate handling for long double functions into a module, for a > platform that doesn't support them. I can't write something in a > Cython module like: > > if sys.platform != 'cygwin': > def my_long_double_func(long double x): > .... No, that's not something we support well. You can put them in a separate module and conditionally compile/import them. > Maybe would be nice to have a simple compiler directive that outright > disables compiling a function depending on a platform check. (I think > this would also be useful for the test framework--the current system > of module-level tags is not fine-grained as I would like, I'm finding. > >>> My suggestion would be to replace the >>> current calls to [trunc,fmod,pow]l to the appropriate >>> __Pyx_%(func)_%(type) wrapper which would provide two alternatives: >>> One that works properly, and one that falls back to using the double >>> version of the function (Numpy does this so there is some precedent). >>> I would go further and suggest throwing up a pre-compiler warning when >>> using the double versions. >>> >>> The only question is what condition (in my mind) is what to use as a >>> condition for selecting the fallbacks. For now I'm thinking something >>> quite specific like >>> >>> #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) >>> >>> since I'm apparently the only person who cares about this at the >>> moment, and Cygwin is the platform I'm targeting. >>> >>> Again, my only real concern right now is that Cython generates code >>> that can be compiled on Cygwin. >> >> >> If you want this behavior, wouldn't including a header >> >> #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) >> #define truncl trunc >> #define fmodl fmod >> #define powl pow >> #endif >> >> as part of your package do the job? > > That's a reasonable workaround. But I would argue that should be baked > into Cythonized code, at least optionally, and should generate a > warning. This is dangerous to bake into Cython code, even with a warning (who would see it?) I say if you want to pretend to support long double on platforms that don't fully support long double, that should be very explicit. From erik.m.bray at gmail.com Wed May 11 09:22:36 2016 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 11 May 2016 15:22:36 +0200 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: References:

Message-ID: On Tue, May 10, 2016 at 8:01 AM, Robert Bradshaw wrote: > On Wed, May 4, 2016 at 8:01 AM, Erik Bray wrote: >> On Tue, May 3, 2016 at 7:15 PM, Robert Bradshaw wrote: >>> On Tue, May 3, 2016 at 3:04 AM, Erik Bray wrote: >>>> I agree that it would be safest not to use long double where long >>>> double functions are not support, and an application that does should >>>> probably check for that. >>>> >>>> The problem is that Cython is generating code that simply *cannot* be >>>> compiled on such platforms. >>> >>> Does Cython ever generate such code when long doubles are not used in the >>> source? If so, we should fix that. But I don't think Cython should be >>> swapping in double versions for long double operations. >> >> No, but the problem is that Cython is a pretty heavy layer of >> abstraction. In the example I gave no long double function was ever >> used explicitly--it was added by a compiler optimization. I probably >> wouldn't write that code thinking that I will need a workaround on >> platforms that don't have truncl. > > I would argue that if you don't have truncl, you shouldn't be using > long double. Possibly this particular use of truncl could be worked > around (though note that using trunc is not safe) but it's starting to > get on shaky ground. Yeah but I will state again: Cython is compiling code that may be distributed to a wide audience. A developer who writes some code that happens to use long double isn't going to think themselves "Gosh, I guess I can't accept `long double` here because it may cause Cython to generate code that contains "truncl" [how would they know that? It's an internal optimization] and I can't do that because it won't work on Cygwin [which the average developer wouldn't know]" Instead they'll just write their code, ship it, and it will fail to compile or install for users on Cygwin. So it's a question of least astonishment. >> In the slightly more explicit cases of fmod and pow I might agree. >> Though there still isn't a great way to anticipate the need to write >> separate handling for long double functions into a module, for a >> platform that doesn't support them. I can't write something in a >> Cython module like: >> >> if sys.platform != 'cygwin': >> def my_long_double_func(long double x): >> .... > > No, that's not something we support well. You can put them in a > separate module and conditionally compile/import them. >> Maybe would be nice to have a simple compiler directive that outright >> disables compiling a function depending on a platform check. (I think >> this would also be useful for the test framework--the current system >> of module-level tags is not fine-grained as I would like, I'm finding. >> >>>> My suggestion would be to replace the >>>> current calls to [trunc,fmod,pow]l to the appropriate >>>> __Pyx_%(func)_%(type) wrapper which would provide two alternatives: >>>> One that works properly, and one that falls back to using the double >>>> version of the function (Numpy does this so there is some precedent). >>>> I would go further and suggest throwing up a pre-compiler warning when >>>> using the double versions. >>>> >>>> The only question is what condition (in my mind) is what to use as a >>>> condition for selecting the fallbacks. For now I'm thinking something >>>> quite specific like >>>> >>>> #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) >>>> >>>> since I'm apparently the only person who cares about this at the >>>> moment, and Cygwin is the platform I'm targeting. >>>> >>>> Again, my only real concern right now is that Cython generates code >>>> that can be compiled on Cygwin. >>> >>> >>> If you want this behavior, wouldn't including a header >>> >>> #if defined(__CYGWIN__) && defined(_LDBL_EQ_DBL) >>> #define truncl trunc >>> #define fmodl fmod >>> #define powl pow >>> #endif >>> >>> as part of your package do the job? >> >> That's a reasonable workaround. But I would argue that should be baked >> into Cythonized code, at least optionally, and should generate a >> warning. > > This is dangerous to bake into Cython code, even with a warning (who > would see it?) I say if you want to pretend to support long double on > platforms that don't fully support long double, that should be very > explicit. Well, something should be explicit anyways. Right now we just get code that doesn't compile on some systems :( I don't necessarily want or care for something to pretend to support long double on a platform where it isn't fully supported. All I care is that my Cython-generated code compiles. I'm just throwing out suggestions, and the possibilities are limited due to lack of support in C for compile-time feature checks. As for who would see such a warning, anyone who compiles the code? I'm talking about preprocessor warnings here. Yes, people ignore warnings, but at least they have something to refer back to if they get incorrect results. Heck, to be safe you're right that it should really be an #error by default, but at least let's make it an explicit error (with the ability to downgrade to a warning). and not "such and such not found". Best, Erik From jdemeyer at cage.ugent.be Wed May 11 16:28:31 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Wed, 11 May 2016 22:28:31 +0200 Subject: [Cython] @staticmethod decorator applied twice Message-ID: <573395EF.2000105@cage.ugent.be> When investigating some issue with decorators, I noticed the following stange thing: it seems that the @staticmethod decorator is actually applied twice in some cases. For example, the testsuite file tests/run/static_methods.pyx contains the following code: cdef class A: @staticmethod def static_def(int x): ... The relevant part of the Cython-generated code: __pyx_t_1 = PyCFunction_NewEx(&__pyx_mdef_14static_methods_1A_1static_def, NULL, __pyx_n_s_static_methods); ... PyTuple_SET_ITEM(__pyx_t_2, 0, __pyx_t_1); __pyx_t_1 = __Pyx_PyObject_Call(__pyx_builtin_staticmethod, __pyx_t_2, NULL); ... if (PyDict_SetItem((PyObject *)__pyx_ptype_14static_methods_A->tp_dict, __pyx_n_s_static_def, __pyx_t_1) < 0) ... __pyx_t_1 = __Pyx_GetNameInClass((PyObject *)__pyx_ptype_14static_methods_A, __pyx_n_s_static_def); ... PyTuple_SET_ITEM(__pyx_t_2, 0, __pyx_t_1); __pyx_t_1 = __Pyx_PyObject_Call(__pyx_builtin_staticmethod, __pyx_t_2, NULL); ... if (PyDict_SetItem((PyObject *)__pyx_ptype_14static_methods_A->tp_dict, __pyx_n_s_static_def, __pyx_t_1) < 0) I might be missing something, but this really looks like Cython is applying the @staticmethod decorator twice. From greg.ewing at canterbury.ac.nz Wed May 11 18:55:21 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 12 May 2016 10:55:21 +1200 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: References:

Message-ID: <5733B859.6020406@canterbury.ac.nz> Erik Bray wrote: > A developer who writes some code that > happens to use long double isn't going to think themselves "Gosh, I > guess I can't accept `long double` here because it may cause Cython to > generate code that contains "truncl" Sounds to me like Cython shouldn't be taking it upon itself to generate calls to truncl just because long double is being used. The programmer should have to call it explicitly if that's what they want. -- Greg From jdemeyer at cage.ugent.be Thu May 12 04:29:37 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Thu, 12 May 2016 10:29:37 +0200 Subject: [Cython] Illegal C code generated with numpy complex type Message-ID: <57343EF1.3090903@cage.ugent.be> Current Cython git master breaks SageMath because Cython generates calls to non-existing functions like fabs_npy_float64(). Since this is a regression since Cython 0.24, I made a blocker ticket http://trac.cython.org/ticket/879 Jeroen. From jdemeyer at cage.ugent.be Thu May 12 09:59:34 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Thu, 12 May 2016 15:59:34 +0200 Subject: [Cython] @staticmethod decorator applied twice In-Reply-To: <573395EF.2000105@cage.ugent.be> References: <573395EF.2000105@cage.ugent.be> Message-ID: <57348C46.1000301@cage.ugent.be> I created http://trac.cython.org/ticket/880 for some issues regarding @staticmethod From stefan_ml at behnel.de Fri May 13 05:05:25 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 May 2016 11:05:25 +0200 Subject: [Cython] missing cimport in module '.' - bug? In-Reply-To: <87lh3rw7fh.fsf@grothesque.org> References: <87lh3rw7fh.fsf@grothesque.org> Message-ID: <573598D5.50705@behnel.de> Christoph Groth schrieb am 03.05.2016 um 16:19: > The setup.py script of our package "kwant" uses Cython.Build.cythonize. > Everything works well, except that for pyx files that cimport a module > from "." an error message is generated. E.g. for the file > "kwant/linalg/lapack.pyx" that contains the line > > from . cimport f_lapack > > The following message appears whenever setup.py is run: > > missing cimport in module '.': kwant/linalg/lapack.pyx > > No error message appears for files that cimport only from somewhere else > than ".", e.g.: > > from .defs cimport gint > > The said messages do not seem to cause any problems, but they are of > course somewhat of a nuiscance. They seem to be created by > Cython.Build.Dependencies.cimported_files(). The culprit seems to be > the function find_pxd in that module that does not seem to handle > properly the case when module is ".". Yes, that's a known problem. It only impacts the dependency collector that decides if a rebuild is necessary, and which is intentionally kept simple. It does not have an effect on the compiler itself. Looks like no-one has worked on this so far, but the correct code is available in other places (obviously) where the imports are resolved. Patches welcome. Stefan From erik.m.bray at gmail.com Fri May 13 07:26:34 2016 From: erik.m.bray at gmail.com (Erik Bray) Date: Fri, 13 May 2016 13:26:34 +0200 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: <5733B859.6020406@canterbury.ac.nz> References:

<5733B859.6020406@canterbury.ac.nz> Message-ID: On Thu, May 12, 2016 at 12:55 AM, Greg Ewing wrote: > Erik Bray wrote: >> >> A developer who writes some code that >> happens to use long double isn't going to think themselves "Gosh, I >> guess I can't accept `long double` here because it may cause Cython to >> generate code that contains "truncl" > > > Sounds to me like Cython shouldn't be taking it upon > itself to generate calls to truncl just because long > double is being used. The programmer should have to > call it explicitly if that's what they want. Since it's just a small optimization it could be disabled without trouble too. Preferably with a #define that can be set or unset at compile time. Best, Erik From yselivanov.ml at gmail.com Sat May 14 17:31:02 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 14 May 2016 17:31:02 -0400 Subject: [Cython] segfault in 'async def' coroutines Message-ID: <57379916.5050403@gmail.com> Hi, Under some circumstances, in asyncio code that runs in uvloop [1], cython code segfaults in cython/Cython/Utility/Coroutine.c: static PyObject * __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) { Py_INCREF(self->gi_qualname); return self->gi_qualname; } "self->gi_qualname" can be NULL. The correct code is probably: __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) { if (self->gi_qualname == NULL) { return __pyx_empty_unicode; } Py_INCREF(self->gi_qualname); return self->gi_qualname; } Thanks, Yury From yselivanov.ml at gmail.com Sat May 14 18:28:29 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 14 May 2016 18:28:29 -0400 Subject: [Cython] async def coroutines miss __module__ Message-ID: <5737A68D.9030107@gmail.com> In CPython: >>> async def foo(): pass >>> foo.__module__ '__main__' In Cython, async def coroutines lack __module__ attribute Yury From yselivanov.ml at gmail.com Sat May 14 19:32:53 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Sat, 14 May 2016 19:32:53 -0400 Subject: [Cython] segfault in 'async def' coroutines In-Reply-To: <57379916.5050403@gmail.com> References: <57379916.5050403@gmail.com> Message-ID: <5737B5A5.4040700@gmail.com> I've just discovered that same thing has to be fixed for __name__. On 2016-05-14 5:31 PM, Yury Selivanov wrote: > Hi, > > Under some circumstances, in asyncio code that runs in uvloop [1], > cython code segfaults in cython/Cython/Utility/Coroutine.c: > > > static PyObject * > __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) > { > Py_INCREF(self->gi_qualname); > return self->gi_qualname; > } > > > "self->gi_qualname" can be NULL. The correct code is probably: > > > __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) > { > if (self->gi_qualname == NULL) { return __pyx_empty_unicode; } > Py_INCREF(self->gi_qualname); > return self->gi_qualname; > } > > > Thanks, > Yury From stefan_ml at behnel.de Mon May 16 03:58:27 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 May 2016 09:58:27 +0200 Subject: [Cython] segfault in 'async def' coroutines In-Reply-To: <57379916.5050403@gmail.com> References: <57379916.5050403@gmail.com> Message-ID: <57397DA3.5080205@behnel.de> Yury Selivanov schrieb am 14.05.2016 um 23:31: > Under some circumstances, in asyncio code that runs in uvloop [1], > cython code segfaults in cython/Cython/Utility/Coroutine.c: > > > static PyObject * > __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) > { > Py_INCREF(self->gi_qualname); > return self->gi_qualname; > } > > > "self->gi_qualname" can be NULL. The correct code is probably: > > > __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) > { > if (self->gi_qualname == NULL) { return __pyx_empty_unicode; } > Py_INCREF(self->gi_qualname); > return self->gi_qualname; > } I wonder why it can be NULL. It's supposed to be set to a string constant at creation time. See GeneratorDefNode in Nodes.py. Is there anything special you are doing with these objects? Could you try to figure out how the ones that have a NULL value here are created? Stefan From stefan_ml at behnel.de Mon May 16 04:47:50 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 May 2016 10:47:50 +0200 Subject: [Cython] async def coroutines miss __module__ In-Reply-To: <5737A68D.9030107@gmail.com> References: <5737A68D.9030107@gmail.com> Message-ID: <57398936.9040406@behnel.de> Yury Selivanov schrieb am 15.05.2016 um 00:28: > In CPython: > >>>> async def foo(): pass >>>> foo.__module__ > '__main__' > > In Cython, async def coroutines lack __module__ attribute Ok, that's easy enough to add. https://github.com/cython/cython/commit/c975662204754a42963ba5b293e3983937615056 Stefan From yselivanov.ml at gmail.com Mon May 16 16:19:05 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 16 May 2016 16:19:05 -0400 Subject: [Cython] segfault in 'async def' coroutines In-Reply-To: <57397DA3.5080205@behnel.de> References: <57379916.5050403@gmail.com> <57397DA3.5080205@behnel.de> Message-ID: <573A2B39.8070303@gmail.com> On 2016-05-16 3:58 AM, Stefan Behnel wrote: > Yury Selivanov schrieb am 14.05.2016 um 23:31: >> Under some circumstances, in asyncio code that runs in uvloop [1], >> cython code segfaults in cython/Cython/Utility/Coroutine.c: >> >> >> static PyObject * >> __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) >> { >> Py_INCREF(self->gi_qualname); >> return self->gi_qualname; >> } >> >> >> "self->gi_qualname" can be NULL. The correct code is probably: >> >> >> __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) >> { >> if (self->gi_qualname == NULL) { return __pyx_empty_unicode; } >> Py_INCREF(self->gi_qualname); >> return self->gi_qualname; >> } > I wonder why it can be NULL. It's supposed to be set to a string constant > at creation time. See GeneratorDefNode in Nodes.py. Is there anything > special you are doing with these objects? Could you try to figure out how > the ones that have a NULL value here are created? It's probably because asyncio Future and Task have __del__ methods which (may) try to call repr on coroutines. At that point of time, maybe the GC breaks the refs. Would you be able to fix the refs for qualname/name? Yury From stefan_ml at behnel.de Tue May 17 01:32:33 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 May 2016 07:32:33 +0200 Subject: [Cython] segfault in 'async def' coroutines In-Reply-To: <573A2B39.8070303@gmail.com> References: <57379916.5050403@gmail.com> <57397DA3.5080205@behnel.de> <573A2B39.8070303@gmail.com> Message-ID: <573AACF1.5060400@behnel.de> Yury Selivanov schrieb am 16.05.2016 um 22:19: > On 2016-05-16 3:58 AM, Stefan Behnel wrote: >> Yury Selivanov schrieb am 14.05.2016 um 23:31: >>> Under some circumstances, in asyncio code that runs in uvloop [1], >>> cython code segfaults in cython/Cython/Utility/Coroutine.c: >>> >>> >>> static PyObject * >>> __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) >>> { >>> Py_INCREF(self->gi_qualname); >>> return self->gi_qualname; >>> } >>> >>> >>> "self->gi_qualname" can be NULL. The correct code is probably: >>> >>> >>> __Pyx_Coroutine_get_qualname(__pyx_CoroutineObject *self) >>> { >>> if (self->gi_qualname == NULL) { return __pyx_empty_unicode; } >>> Py_INCREF(self->gi_qualname); >>> return self->gi_qualname; >>> } >> I wonder why it can be NULL. It's supposed to be set to a string constant >> at creation time. See GeneratorDefNode in Nodes.py. Is there anything >> special you are doing with these objects? Could you try to figure out how >> the ones that have a NULL value here are created? > > It's probably because asyncio Future and Task have __del__ methods > which (may) try to call repr on coroutines. At that point of time, > maybe the GC breaks the refs. > > Would you be able to fix the refs for qualname/name? It seems that CPython should suffer from the same problem, though. It uses the same code in the getter. Also, should we really return an empty string or None? Both feel like unexpected results, so I can't tell which is better. And finally, since both name values are guaranteed to be strings (the setter checks their type), I wonder if we shouldn't just make sure they are *exactly* Unicode strings by converting any subtypes, and then remove their Py_CLEAR() from the tp_clear() function. Strings can't participate in reference cycles, and we could thus report the actual names even during garbage collection. Stefan From yselivanov.ml at gmail.com Tue May 17 11:31:06 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Tue, 17 May 2016 11:31:06 -0400 Subject: [Cython] segfault in 'async def' coroutines In-Reply-To: <573AACF1.5060400@behnel.de> References: <57379916.5050403@gmail.com> <57397DA3.5080205@behnel.de> <573A2B39.8070303@gmail.com> <573AACF1.5060400@behnel.de> Message-ID: <573B393A.6020904@gmail.com> On 2016-05-17 1:32 AM, Stefan Behnel wrote: > And finally, since both name values are guaranteed to be strings (the > setter checks their type), I wonder if we shouldn't just make sure they are > *exactly* Unicode strings by converting any subtypes, and then remove their > Py_CLEAR() from the tp_clear() function. Strings can't participate in > reference cycles, and we could thus report the actual names even during > garbage collection. This sounds like a really nice solution, Stefan. Yury From robertwb at gmail.com Tue May 17 18:22:45 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 17 May 2016 15:22:45 -0700 Subject: [Cython] Cygwin: Handling missing C99 long double functions In-Reply-To: References:

<5733B859.6020406@canterbury.ac.nz> Message-ID: On Fri, May 13, 2016 at 4:26 AM, Erik Bray wrote: > On Thu, May 12, 2016 at 12:55 AM, Greg Ewing > wrote: >> Erik Bray wrote: >>> >>> A developer who writes some code that >>> happens to use long double isn't going to think themselves "Gosh, I >>> guess I can't accept `long double` here because it may cause Cython to >>> generate code that contains "truncl" >> >> >> Sounds to me like Cython shouldn't be taking it upon >> itself to generate calls to truncl just because long >> double is being used. The programmer should have to >> call it explicitly if that's what they want. > > Since it's just a small optimization it could be disabled without > trouble too. Preferably with a #define that can be set or unset at > compile time. https://github.com/cython/cython/commit/46d3efc4ff9123c73889bcb54b2b200d6be39ff4 From yselivanov.ml at gmail.com Thu May 19 11:29:05 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 19 May 2016 11:29:05 -0400 Subject: [Cython] segfault in 'async def' coroutines In-Reply-To: <573B393A.6020904@gmail.com> References: <57379916.5050403@gmail.com> <57397DA3.5080205@behnel.de> <573A2B39.8070303@gmail.com> <573AACF1.5060400@behnel.de> <573B393A.6020904@gmail.com> Message-ID: <573DDBC1.3010409@gmail.com> Stefan, any ETA on this? For now I have to patch the C file generated by Cython (otherwise uvloop segfaults), and that's kind of fragile. Yury On 2016-05-17 11:31 AM, Yury Selivanov wrote: > > > On 2016-05-17 1:32 AM, Stefan Behnel wrote: >> And finally, since both name values are guaranteed to be strings (the >> setter checks their type), I wonder if we shouldn't just make sure >> they are >> *exactly* Unicode strings by converting any subtypes, and then >> remove their >> Py_CLEAR() from the tp_clear() function. Strings can't participate in >> reference cycles, and we could thus report the actual names even during >> garbage collection. > > This sounds like a really nice solution, Stefan. > > Yury From stefan_ml at behnel.de Thu May 19 12:28:30 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 May 2016 18:28:30 +0200 Subject: [Cython] segfault in 'async def' coroutines In-Reply-To: <573DDBC1.3010409@gmail.com> References: <57379916.5050403@gmail.com> <57397DA3.5080205@behnel.de> <573A2B39.8070303@gmail.com> <573AACF1.5060400@behnel.de> <573B393A.6020904@gmail.com> <573DDBC1.3010409@gmail.com> Message-ID: <573DE9AE.9090305@behnel.de> Yury Selivanov schrieb am 19.05.2016 um 17:29: > On 2016-05-17 11:31 AM, Yury Selivanov wrote: >> On 2016-05-17 1:32 AM, Stefan Behnel wrote: >>> And finally, since both name values are guaranteed to be strings (the >>> setter checks their type), I wonder if we shouldn't just make sure they are >>> *exactly* Unicode strings by converting any subtypes, and then remove >>> their >>> Py_CLEAR() from the tp_clear() function. Strings can't participate in >>> reference cycles, and we could thus report the actual names even during >>> garbage collection. >> >> This sounds like a really nice solution, Stefan. > > Stefan, any ETA on this? For now I have to patch the C file generated by > Cython (otherwise uvloop segfaults), and that's kind of fragile. Could you give it a try yourself? The code is in Coroutine.c, should be straight forward. Stefan From jdemeyer at cage.ugent.be Tue May 31 06:28:43 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Tue, 31 May 2016 12:28:43 +0200 Subject: [Cython] Supporting cython.operator in C Message-ID: <574D675B.1040207@cage.ugent.be> The following code cimport cython.operator cdef long foo(long x, long y): return cython.operator.comma(x, y) gives AttributeError: 'CBinopNode' object has no attribute 'analyse_c_operation' Is there any fundamental reason why such operators are only implemented for C++ and not pure C? What would be needed to support such operators in C? Jeroen. From robertwb at gmail.com Tue May 31 14:35:43 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 31 May 2016 11:35:43 -0700 Subject: [Cython] Supporting cython.operator in C In-Reply-To: <574D675B.1040207@cage.ugent.be> References: <574D675B.1040207@cage.ugent.be> Message-ID: I can't think of any fundamental reason these couldn't be implemented in C, but there's really no need to do so as they can't be overridden in C. On Tue, May 31, 2016 at 3:28 AM, Jeroen Demeyer wrote: > The following code > > cimport cython.operator > cdef long foo(long x, long y): > return cython.operator.comma(x, y) > > gives AttributeError: 'CBinopNode' object has no attribute > 'analyse_c_operation' > > Is there any fundamental reason why such operators are only implemented for > C++ and not pure C? What would be needed to support such operators in C? > > > Jeroen. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel