From zpincus at gmail.com Sun Mar 6 17:58:54 2016 From: zpincus at gmail.com (Zach Pincus) Date: Sun, 6 Mar 2016 16:58:54 -0600 Subject: [Cython] segfault with fused types and __kwdefaults__ Message-ID: Hi all, Just found an odd bug, which occurs with both Python 3.4.3 / Cython 0.23.4 and Python 2.7.10 / Cython 0.22.1 (the versions that happen to be on my machine). Specifically, a cython function with a fused type *and* a keyword argument in its signature will cause a segfault when its __kwdefaults__ attribute is looked up. (Which happens if, say, you try to get the function documentation in python via "function?", or otherwise do any inspection...) Example: foo.pyx ----- ctypedef fused FUSED: float int def foo(FUSED bar, baz=None): pass ----- in python: >>> import pyximport; pyximport.install() (None, ) >>> import foo >>> foo.foo.__kwdefaults__ zsh: segmentation fault This works of course when compiled with a setup.py script as well; the above is just the most minimal test case. There's no segfault without the keyword argument, and no segfault if the fused type is replaced with a regular type. Here's the chunk of the backtrace that's in cython code (from running python3 under lldb): * frame #0: 0x0000000103b801b9 foo.so`__pyx_pf_3foo_12__defaults__(__pyx_self=0x0000000102741458) + 41 at foo.c:1294 1291 __Pyx_XDECREF(__pyx_r); 1292 __pyx_t_1 = PyTuple_New(1); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 5; __pyx_clineno = __LINE__; goto __pyx_L1_error;} 1293 __Pyx_GOTREF(__pyx_t_1); -> 1294 __Pyx_INCREF(__Pyx_CyFunction_Defaults(__pyx_defaults2, __pyx_self)->__pyx_arg_baz); 1295 __Pyx_GIVEREF(__Pyx_CyFunction_Defaults(__pyx_defaults2, __pyx_self)->__pyx_arg_baz); 1296 PyTuple_SET_ITEM(__pyx_t_1, 0, __Pyx_CyFunction_Defaults(__pyx_defaults2, __pyx_self)->__pyx_arg_baz); 1297 __pyx_t_2 = PyTuple_New(2); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 5; __pyx_clineno = __LINE__; goto __pyx_L1_error;} frame #1: 0x0000000103b838c0 foo.so`__Pyx_CyFunction_get_kwdefaults [inlined] __Pyx_CyFunction_init_defaults(op=0x0000000102741458) + 5 at foo.c:2789 frame #2: 0x0000000103b838bb foo.so`__Pyx_CyFunction_get_kwdefaults(op=0x0000000102741458) + 43 at foo.c:2859 Best, Zach Zachary Pincus, PhD Assistant Professor of Developmental Biology and Genetics Washington University School of Medicine http://zplab.wustl.edu From matthew.brett at gmail.com Mon Mar 7 20:47:32 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 7 Mar 2016 17:47:32 -0800 Subject: [Cython] Manylinux wheels for Cython Message-ID: Hi, I don't know whether y'all have been following over at distutils-sig, but there's a new distutils PEP that defines a `manylinux` format for Linux wheels that work on many different x86 Linux distributions: https://www.python.org/dev/peps/pep-0513/ https://github.com/pypa/manylinux The latest version of pip will install these, if the client Linux system is compatible with the manylinux spec: https://pip.pypa.io/en/stable/news/ I've already built and used manylinux Cython wheels, which y'all are welcome to test with: pip install -f https://nipy.bic.berkeley.edu/manylinux cython (The wheels there don't have the right manylinux filenames yet, but they have the same contents as the ones that would go up to pypi). I've already had good use from these wheels in speeding up project builds into docker containers and virtualenvs, and I'd love to upload these to pypi. I have permissions on pypi to do this, but I wanted to check in with y'all first... Cheers, Matthew From ian.h.bell at gmail.com Tue Mar 8 01:28:38 2016 From: ian.h.bell at gmail.com (Ian Bell) Date: Mon, 7 Mar 2016 23:28:38 -0700 Subject: [Cython] Manylinux wheels for Cython In-Reply-To: References: Message-ID: Jorrit, maybe in interesting for us? On Mar 7, 2016 6:48 PM, "Matthew Brett" wrote: > Hi, > > I don't know whether y'all have been following over at distutils-sig, > but there's a new distutils PEP that defines a `manylinux` format for > Linux wheels that work on many different x86 Linux distributions: > > https://www.python.org/dev/peps/pep-0513/ > https://github.com/pypa/manylinux > > The latest version of pip will install these, if the client Linux > system is compatible with the manylinux spec: > > https://pip.pypa.io/en/stable/news/ > > I've already built and used manylinux Cython wheels, which y'all are > welcome to test with: > > pip install -f https://nipy.bic.berkeley.edu/manylinux cython > > (The wheels there don't have the right manylinux filenames yet, but > they have the same contents as the ones that would go up to pypi). > > I've already had good use from these wheels in speeding up project > builds into docker containers and virtualenvs, and I'd love to upload > these to pypi. I have permissions on pypi to do this, but I wanted > to check in with y'all first... > > Cheers, > > Matthew > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Fri Mar 11 03:23:53 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 11 Mar 2016 00:23:53 -0800 Subject: [Cython] Cython alpha 0.24 Message-ID: We are getting ready to push a new major release. A first alpha is up at http://cython.org/release/Cython-0.24a0.tar.gz To avoid surprises, please test and report. Thanks, Robert From jdemeyer at cage.ugent.be Fri Mar 11 03:30:34 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Fri, 11 Mar 2016 09:30:34 +0100 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: Message-ID: <56E2822A.3080105@cage.ugent.be> Hello, Is there any chance of merging https://github.com/cython/cython/pull/483 It is a really trivial patch. Thanks, Jeroen. From robertwb at gmail.com Fri Mar 11 04:00:43 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 11 Mar 2016 01:00:43 -0800 Subject: [Cython] Cython alpha 0.24 In-Reply-To: <56E2822A.3080105@cage.ugent.be> References: <56E2822A.3080105@cage.ugent.be> Message-ID: I'd really rather deprecate .pxi files completely rather than add even more support for them. I haven't dived into cysignals yet, but it seems it could be done with a cimport alone. On Fri, Mar 11, 2016 at 12:30 AM, Jeroen Demeyer wrote: > Hello, > > Is there any chance of merging https://github.com/cython/cython/pull/483 > It is a really trivial patch. > > Thanks, > Jeroen. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From jdemeyer at cage.ugent.be Fri Mar 11 04:13:34 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Fri, 11 Mar 2016 10:13:34 +0100 Subject: [Cython] Support for .pxi files Message-ID: <56E28C3E.2080106@cage.ugent.be> Dear Cython developers, I'm writing this in response to Robert's "I'd really rather deprecate .pxi files completely". Can you please explain what you have against .pxi files? I know they have been abused and 95% of the historic use cases should be replaced by other means. However, that alone is not sufficient reason to stop supporting them: there is still the 5% of cases where they are really useful. I think that you should continue to support .pxi files on a same level as .pxd files. I still do not understand why https://github.com/cython/cython/pull/392 was refused and it seems that https://github.com/cython/cython/pull/483 is going to be refused too. Jeroen Demeyer. From jdemeyer at cage.ugent.be Fri Mar 11 04:15:36 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Fri, 11 Mar 2016 10:15:36 +0100 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: <56E2822A.3080105@cage.ugent.be> Message-ID: <56E28CB8.1020205@cage.ugent.be> On 2016-03-11 10:00, Robert Bradshaw wrote: > but it seems it could be done with a cimport alone. I already discussed that with Stefan Behnel... no, it cannot be done with a cimport alone. From robertwb at gmail.com Fri Mar 11 04:34:33 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 11 Mar 2016 01:34:33 -0800 Subject: [Cython] Support for .pxi files In-Reply-To: <56E28C3E.2080106@cage.ugent.be> References: <56E28C3E.2080106@cage.ugent.be> Message-ID: On Fri, Mar 11, 2016 at 1:13 AM, Jeroen Demeyer wrote: > Dear Cython developers, > > I'm writing this in response to Robert's "I'd really rather deprecate .pxi > files completely". > > Can you please explain what you have against .pxi files? They're neither necessary nor pythonic. Even C++ is trying to move away from #includes to proper modules. On a more pragmatic note, includes add complexities to dependency tracking (among other things) and we still have cruft from abuse of pxi files, e.g. https://github.com/cython/cython/blob/16ad9e21da64c4de4f964980ad1b8c18ca70035b/Cython/Compiler/ExprNodes.py#L891 > I know they have > been abused and 95% of the historic use cases should be replaced by other > means. However, that alone is not sufficient reason to stop supporting them: > there is still the 5% of cases where they are really useful. Every single case I can think of would be better done by using cimports or preprocessing via a proper templating (or at least preprocessing) utility. Now cysignals is the kind of package that might be doing something tricky that makes this a special case, but even then I'd call that 0.5% not 5%. I need to dive into that deeper. > I think that you should continue to support .pxi files on a same level as > .pxd files. I still do not understand why > https://github.com/cython/cython/pull/392 was refused and it seems that > https://github.com/cython/cython/pull/483 is going to be refused too. > > Jeroen Demeyer. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From jdemeyer at cage.ugent.be Fri Mar 11 04:58:10 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Fri, 11 Mar 2016 10:58:10 +0100 Subject: [Cython] Support for .pxi files In-Reply-To: References: <56E28C3E.2080106@cage.ugent.be> Message-ID: <56E296B2.1090204@cage.ugent.be> On 2016-03-11 10:34, Robert Bradshaw wrote: > On a more pragmatic note, includes add complexities to dependency > tracking Because you *intentionally* make things complicated. The refused pull request 392 could actually make dependency checking simpler by treating .pxd and .pxi files more alike. > Every single case I can think of would be better done by using > cimports or preprocessing via a proper templating (or at least > preprocessing) utility. The difference is that such preprocessing will be external to Cython, which will make the build process more complex. You need to figure out what to do with the files generated by the templating and you need to get dependency checking right... I must admit that I am quite happy that Cython solves these problems for me. Do you have an example of a project which actually uses such templating/preprocessing? Jeroen. From jdemeyer at cage.ugent.be Fri Mar 11 05:09:16 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Fri, 11 Mar 2016 11:09:16 +0100 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: Message-ID: <56E2994C.9080702@cage.ugent.be> I get one testsuite failure: $ python runtests.py -v -v fused_cpp Python 2.7.10 (default, Mar 11 2016, 09:57:44) [GCC 4.8.4] Running tests against Cython 0.24a0 Backends: c,cpp runTest (__main__.CythonRunTestCase) compiling (cpp) and running fused_cpp ... === Expected errors: === === Got errors: === 14:16: Compiler crash in AnalyseDeclarationsTransform ERROR ====================================================================== ERROR: runTest (__main__.CythonRunTestCase) compiling (cpp) and running fused_cpp ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 993, in run ext_so_path = self.runCompileTest() File "runtests.py", line 705, in runCompileTest self.test_directory, self.expect_errors, self.annotate) File "runtests.py", line 928, in compile self.assertEquals(None, unexpected_error) AssertionError: None != u'14:16: Compiler crash in AnalyseDeclarationsTransform' ---------------------------------------------------------------------- Ran 1 test in 0.350s FAILED (errors=1) ALL DONE From robertwb at gmail.com Fri Mar 11 05:24:11 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 11 Mar 2016 02:24:11 -0800 Subject: [Cython] Cython alpha 0.24 In-Reply-To: <56E2994C.9080702@cage.ugent.be> References: <56E2994C.9080702@cage.ugent.be> Message-ID: https://github.com/cython/cython/commit/970c2fc0e676ca22016e14147ada0edba937dc6b (Just noticed the build failed on travis too...) On Fri, Mar 11, 2016 at 2:09 AM, Jeroen Demeyer wrote: > I get one testsuite failure: > > $ python runtests.py -v -v fused_cpp > Python 2.7.10 (default, Mar 11 2016, 09:57:44) > [GCC 4.8.4] > > Running tests against Cython 0.24a0 > Backends: c,cpp > > runTest (__main__.CythonRunTestCase) > compiling (cpp) and running fused_cpp ... > === Expected errors: === > > > === Got errors: === > 14:16: Compiler crash in AnalyseDeclarationsTransform > > ERROR > > ====================================================================== > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running fused_cpp > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "runtests.py", line 993, in run > ext_so_path = self.runCompileTest() > File "runtests.py", line 705, in runCompileTest > self.test_directory, self.expect_errors, self.annotate) > File "runtests.py", line 928, in compile > self.assertEquals(None, unexpected_error) > AssertionError: None != u'14:16: Compiler crash in > AnalyseDeclarationsTransform' > > ---------------------------------------------------------------------- > Ran 1 test in 0.350s > > FAILED (errors=1) > ALL DONE > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From krthornt at uci.edu Fri Mar 11 18:53:36 2016 From: krthornt at uci.edu (Kevin Thornton) Date: Fri, 11 Mar 2016 23:53:36 +0000 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: Message-ID: Hi, I just built a large project with the alpha version. All looks good. The only new/odd thing is a lot of warnings of this flavor: warning: View.MemoryView:219:4: 'property memview:' syntax is deprecated, use '@property' warning: View.MemoryView:536:4: 'property T:' syntax is deprecated, use '@property' warning: View.MemoryView:543:4: 'property base:' syntax is deprecated, use '@property' warning: View.MemoryView:548:4: 'property shape:' syntax is deprecated, use '@property' warning: View.MemoryView:553:4: 'property strides:' syntax is deprecated, use '@property' warning: View.MemoryView:562:4: 'property suboffsets:' syntax is deprecated, use '@property' warning: View.MemoryView:570:4: 'property ndim:' syntax is deprecated, use '@property' warning: View.MemoryView:575:4: 'property itemsize:' syntax is deprecated, use '@property' warning: View.MemoryView:580:4: 'property nbytes:' syntax is deprecated, use '@property' warning: View.MemoryView:585:4: 'property size:' syntax is deprecated, use '@property' warning: View.MemoryView:984:4: 'property base:' syntax is deprecated, use '@property' warning: View.MemoryView:219:4: 'property memview:' syntax is deprecated, use '@property' warning: View.MemoryView:536:4: 'property T:' syntax is deprecated, use '@property' warning: View.MemoryView:543:4: 'property base:' syntax is deprecated, use '@property' warning: View.MemoryView:548:4: 'property shape:' syntax is deprecated, use '@property' warning: View.MemoryView:553:4: 'property strides:' syntax is deprecated, use '@property' warning: View.MemoryView:562:4: 'property suboffsets:' syntax is deprecated, use '@property' warning: View.MemoryView:570:4: 'property ndim:' syntax is deprecated, use '@property' warning: View.MemoryView:575:4: 'property itemsize:' syntax is deprecated, use '@property' warning: View.MemoryView:580:4: 'property nbytes:' syntax is deprecated, use '@property' --Kevin On Fri, Mar 11, 2016 at 12:26 AM Robert Bradshaw wrote: > We are getting ready to push a new major release. A first alpha is up > at http://cython.org/release/Cython-0.24a0.tar.gz > > To avoid surprises, please test and report. > > Thanks, > Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdemeyer at cage.ugent.be Sun Mar 13 05:49:24 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Sun, 13 Mar 2016 10:49:24 +0100 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: <56E2822A.3080105@cage.ugent.be> Message-ID: <56E537A4.2090202@cage.ugent.be> On 2016-03-11 10:00, Robert Bradshaw wrote: > I'd really rather deprecate .pxi files completely rather than add even > more support for them. Well, .pxi files are currently *not* deprecated. The fact that they might be deprecated in the future seems a poor excuse to reject a 1-line patch fixing an actual use case. Really, what is the disadvantage to merging that pull request? Jeroen. From robertwb at gmail.com Sun Mar 13 09:03:08 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Sun, 13 Mar 2016 06:03:08 -0700 Subject: [Cython] Cython alpha 0.24 In-Reply-To: <56E537A4.2090202@cage.ugent.be> References: <56E2822A.3080105@cage.ugent.be> <56E537A4.2090202@cage.ugent.be> Message-ID: On Sun, Mar 13, 2016 at 1:49 AM, Jeroen Demeyer wrote: > On 2016-03-11 10:00, Robert Bradshaw wrote: >> >> I'd really rather deprecate .pxi files completely rather than add even >> more support for them. > > > Well, .pxi files are currently *not* deprecated. The fact that they might be > deprecated in the future seems a poor excuse to reject a 1-line patch fixing > an actual use case. Really, what is the disadvantage to merging that pull > request? Right now, at least, pxi files are completely self-contained within a project. If we accept cross-project includes, this will be much more painful to remove in the future as a library now has to fix all its "callers." From robertwb at math.washington.edu Sun Mar 13 09:09:19 2016 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 13 Mar 2016 06:09:19 -0700 Subject: [Cython] Support for .pxi files In-Reply-To: <56E296B2.1090204@cage.ugent.be> References: <56E28C3E.2080106@cage.ugent.be> <56E296B2.1090204@cage.ugent.be> Message-ID: On Fri, Mar 11, 2016 at 1:58 AM, Jeroen Demeyer wrote: > On 2016-03-11 10:34, Robert Bradshaw wrote: >> >> On a more pragmatic note, includes add complexities to dependency >> tracking > > > Because you *intentionally* make things complicated. The refused pull > request 392 could actually make dependency checking simpler by treating .pxd > and .pxi files more alike. We can't because of how relative imports are resolved in included files (they're relative to the includer). This goes way back to the beginning, and would be non-trivial to change. >> Every single case I can think of would be better done by using >> cimports or preprocessing via a proper templating (or at least >> preprocessing) utility. > > > The difference is that such preprocessing will be external to Cython, which > will make the build process more complex. You need to figure out what to do > with the files generated by the templating and you need to get dependency > checking right... I must admit that I am quite happy that Cython solves > these problems for me. > > Do you have an example of a project which actually uses such > templating/preprocessing? Sage (e.g. fast callables). We should make support for this better, e.g. one specify a Python callable or a command line (not sure about cross-platform dependence on that last one) to use for preprocessing right in the source file. However, given that module lists are code, it doesn't seem that hard to generate sources programmatically right before calling cythonize, adding another intermediate into the source -> .so chain. From jpe at wingware.com Tue Mar 15 15:56:35 2016 From: jpe at wingware.com (John Ehresman) Date: Tue, 15 Mar 2016 15:56:35 -0400 Subject: [Cython] Function calls and ref counts Message-ID: <56E868F3.2060303@wingware.com> Hi, I ran into a bug where a python object was deleted in the middle of a function that was using it because a callback decref'd the one and only reference to the object. This doesn't happen with python bytecode because ref counts of the callable and of all args are essentially incref'd before the call and then decref'd after. Doing the same in Cython generated C code might be the way to fix this. The pure python mode code below reproduces the crash with cython 0.23 when compiled to C and the Crash function called. Thanks, John import cython @cython.cclass class Record: cython.declare(ref=object) def setref(self, ref): self.ref = ref GLOBAL_RECORD = Record() @cython.cclass class CrashCls: def method(self): cython.declare(rec=Record) rec = GLOBAL_RECORD print id(self) rec.ref = None assert isinstance(self, CrashCls) # <-- should crash def Crash(): cython.declare(rec=Record) rec = GLOBAL_RECORD o = CrashCls() rec.ref = o.method del o args = () rec.ref(*args) From jdemeyer at cage.ugent.be Tue Mar 15 17:55:56 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Tue, 15 Mar 2016 22:55:56 +0100 Subject: [Cython] Function calls and ref counts In-Reply-To: <56E868F3.2060303@wingware.com> References: <56E868F3.2060303@wingware.com> Message-ID: <56E884EC.1080302@cage.ugent.be> I cannot reproduce the problem... It might be relevant to mention things like OS and Python version. From matthew.brett at gmail.com Tue Mar 15 17:58:36 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 15 Mar 2016 14:58:36 -0700 Subject: [Cython] Manylinux wheels for Cython In-Reply-To: References: Message-ID: Hi, On Mon, Mar 7, 2016 at 5:47 PM, Matthew Brett wrote: > Hi, > > I don't know whether y'all have been following over at distutils-sig, > but there's a new distutils PEP that defines a `manylinux` format for > Linux wheels that work on many different x86 Linux distributions: > > https://www.python.org/dev/peps/pep-0513/ > https://github.com/pypa/manylinux > > The latest version of pip will install these, if the client Linux > system is compatible with the manylinux spec: > > https://pip.pypa.io/en/stable/news/ > > I've already built and used manylinux Cython wheels, which y'all are > welcome to test with: > > pip install -f https://nipy.bic.berkeley.edu/manylinux cython > > (The wheels there don't have the right manylinux filenames yet, but > they have the same contents as the ones that would go up to pypi). > > I've already had good use from these wheels in speeding up project > builds into docker containers and virtualenvs, and I'd love to upload > these to pypi. I have permissions on pypi to do this, but I wanted > to check in with y'all first... There is now a test wheel for Cython 0.23.4 and Python 3.5 on the testpypi server. This is me downloading and installing - a matter of a few seconds: $ python -m pip install -U pip Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-8.1.0-py2.py3-none-any.whl#md5=c6eca6736b2b8f7280fb25e44be7c51b Downloading pip-8.1.0-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded Installing collected packages: pip Found existing installation: pip 1.5.6 Uninstalling pip: Successfully uninstalled pip Successfully installed pip Cleaning up... $ pip install -i https://testpypi.python.org/pypi/ cython Collecting cython Using cached https://testpypi.python.org/packages/cp35/C/Cython/Cython-0.23.4-cp35-cp35m-manylinux1_x86_64.whl Installing collected packages: cython Successfully installed cython-0.23.4 $ cython --version Cython version 0.23.4 The installed Cython version compiles all the Demo *.pyx files OK. See also : https://mail.python.org/pipermail/wheel-builders/2016-March/000050.html Best, Matthew From jdemeyer at cage.ugent.be Tue Mar 15 18:13:53 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Tue, 15 Mar 2016 23:13:53 +0100 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: <56E2822A.3080105@cage.ugent.be> <56E537A4.2090202@cage.ugent.be> Message-ID: <56E88921.3060403@cage.ugent.be> On 2016-03-13 14:03, Robert Bradshaw wrote: > If we accept cross-project includes I missed the memo that cross-project includes are not "accepted". They do work, just add include_path=sys.path to the cythonize() call. All I'm doing with my patch is to remove the need to explicitly add that in setup.py. From jpe at wingware.com Wed Mar 16 11:29:51 2016 From: jpe at wingware.com (John Ehresman) Date: Wed, 16 Mar 2016 11:29:51 -0400 Subject: [Cython] Function calls and ref counts In-Reply-To: <56E884EC.1080302@cage.ugent.be> References: <56E868F3.2060303@wingware.com> <56E884EC.1080302@cage.ugent.be> Message-ID: <56E97BEF.2050208@wingware.com> On 3/15/16 5:55 PM, Jeroen Demeyer wrote: > I cannot reproduce the problem... > > It might be relevant to mention things like OS and Python version. It may only crash in a pydebug build since freed memory isn't necessarily overwritten right away. I tested with a pydebug build of Python 2.7 on OS X. Thanks, John From stefan_ml at behnel.de Sat Mar 19 06:11:46 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 19 Mar 2016 11:11:46 +0100 Subject: [Cython] PEP 515: underscores in number literals Message-ID: <56ED25E2.9080103@behnel.de> Hi! PEP 515 proposes to allow underscores in number literals to provide a visual digit separator, e.g. for thousands, nibbles, hex-bytes/words, etc. https://www.python.org/dev/peps/pep-0515/ I've implemented it for Cython: https://github.com/cython/cython/pull/499 I think it's safe enough to still go into 0.24 as it only extends the supported syntax, but I'll leave it to Robert to decide. Stefan From joshua.adelman at gmail.com Fri Mar 18 11:55:14 2016 From: joshua.adelman at gmail.com (Joshua) Date: Fri, 18 Mar 2016 08:55:14 -0700 (PDT) Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: Message-ID: <28f1bbeb-599e-4685-a2c4-8de0df6a4ae7@googlegroups.com> On Friday, March 11, 2016 at 3:24:26 AM UTC-5, Robert Bradshaw wrote: > > We are getting ready to push a new major release. A first alpha is up > at http://cython.org/release/Cython-0.24a0.tar.gz > > To avoid surprises, please test and report. > > Thanks, > Robert > I ran across a compilation error using 0.24a1. It's actually the same error that I posted to the list a while back (https://groups.google.com/d/msg/cython-users/_ShPVfZAuKA/RxiS0dfRDwAJ), but at the time, I didn't have a minimal working example that had been extracted from a much more complicated project that I couldn't post publicly. I now have a minimal demonstration that I've posted here: https://gist.github.com/synapticarbors/c6268e25ce22e4d04b4d It fails on both OSX and Windows (using mingw) with traceback: $ python setup.py build_ext -i Compiling factors.pyx because it changed. Compiling xlib.pyx because it changed. [1/2] Cythonizing factors.pyx [2/2] Cythonizing xlib.pyx running build_ext building 'factors' extension creating build creating build/temp.macosx-10.5-x86_64-2.7 gcc -fno-strict-aliasing -I/Users/lev/anaconda/envs/test-cython/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/Users/lev/ anaconda/envs/test-cython/lib/python2.7/site-packages/numpy/core/include -I/ Users/lev/anaconda/envs/test-cython/include/python2.7 -c factors.c -o build/ temp.macosx-10.5-x86_64-2.7/factors.o -O3 -w gcc -bundle -undefined dynamic_lookup -L/Users/lev/anaconda/envs/test-cython /lib -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/factors.o - L/Users/lev/anaconda/envs/test-cython/lib -o /Users/lev/Desktop/av4tmp/ cython-test/factors.so building 'xlib' extension gcc -fno-strict-aliasing -I/Users/lev/anaconda/envs/test-cython/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -I/Users/lev/ anaconda/envs/test-cython/lib/python2.7/site-packages/numpy/core/include -I/ Users/lev/anaconda/envs/test-cython/include/python2.7 -c xlib.c -o build/ temp.macosx-10.5-x86_64-2.7/xlib.o -O3 -w xlib.c:17371:63: error: too few arguments to function call, expected 3, have 1 __pyx_v_7factors_xarr0 = __Pyx_carray_from_py_xarr__type(o); if ( PyErr_Occurred()) __PYX_ERR(1, 13, __pyx_L2_error); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ xlib.c:4726:1: note: '__Pyx_carray_from_py_xarr__type' declared here static int __Pyx_carray_from_py_xarr__type(PyObject *__pyx_v_o, struct __pyx_t_15data_structures_xarr_type *__py... ^ xlib.c:20424:13: error: static declaration of '__Pyx_BufFmt_Init' follows non-static declaration static void __Pyx_BufFmt_Init(__Pyx_BufFmt_Context* ctx, ^ xlib.c:20295:9: note: previous implicit declaration is here __Pyx_BufFmt_Init(&ctx, stack, dtype); ^ xlib.c:20789:20: error: static declaration of '__Pyx_BufFmt_CheckString' follows non-static declaration static const char* __Pyx_BufFmt_CheckString(__Pyx_BufFmt_Context* ctx, const char* ts) { ^ xlib.c:20296:14: note: previous implicit declaration is here if (!__Pyx_BufFmt_CheckString(&ctx, buf->format)) goto fail; ^ 3 errors generated. error: command 'gcc' failed with exit status 1 It fails similarly with Cython 0.23.4. It compiles without error using Cython 0.22.1, which is what I'm using in production. It looks like a possible codegen error to me since the function call doesn't match the prototype, but it's deep enough in the cython internals that I'll defer to a Cython developer. Any help with this would be very much appreciated. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.madden at nextthought.com Sat Mar 19 10:44:29 2016 From: jason.madden at nextthought.com (Jason Madden) Date: Sat, 19 Mar 2016 09:44:29 -0500 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: Message-ID: gevent compiles and passes all its tests with this release. However, compilation produces 420 warnings of the form: warning: gevent/corecext.pyx:1893:4: 'property rpid:' syntax is deprecated, use '@property' That's a lot of warnings. As I understand it, this was added with pull #462 which is only in master/0.24, so there On Fri, Mar 11, 2016 at 2:23 AM, Robert Bradshaw wrote: > We are getting ready to push a new major release. A first alpha is up > at http://cython.org/release/Cython-0.24a0.tar.gz > > To avoid surprises, please test and report. > > Thanks, > Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jason.madden at nextthought.com Sat Mar 19 10:51:12 2016 From: jason.madden at nextthought.com (Jason Madden) Date: Sat, 19 Mar 2016 09:51:12 -0500 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: Message-ID: [Ack, really sorry for the premature send. Here's the whole thing:] gevent compiles and passes all its tests with this release. However, compilation produces 420 warnings of the form: warning: gevent/corecext.pyx:1893:4: 'property rpid:' syntax is deprecated, use '@property' That's a lot of warnings. There's also this comment in the code, which seems to indicate that the warnings aren't actually desired, but the test for it includes `if True:` so warnings are always emitted. I'm a bit confused as to the intent; are the warnings going to be in the final release? def visit_PropertyNode(self, node): # Suppress warning for our code until we can convert all our uses over. if isinstance(node.pos[0], str) or True: warning(node.pos, "'property %s:' syntax is deprecated, use '@property'" % node.name, 2) As I understand it, this was added with pull #462 which is only in master/0.24, so there's no way for me to fix these warnings while still being able to compile on 0.23; is that right? (That is, @property doesn't work right in 0.23) If so, this would seem to at least merit a mention in CHANGES.rst? Thanks, jason On Sat, Mar 19, 2016 at 9:44 AM, Jason Madden wrote: > gevent compiles and passes all its tests with this release. > > However, compilation produces 420 warnings of the form: > > warning: gevent/corecext.pyx:1893:4: 'property rpid:' syntax is > deprecated, use '@property' > > That's a lot of warnings. > > As I understand it, this was added with pull #462 which is only in > master/0.24, so there > > > On Fri, Mar 11, 2016 at 2:23 AM, Robert Bradshaw > wrote: > >> We are getting ready to push a new major release. A first alpha is up >> at http://cython.org/release/Cython-0.24a0.tar.gz >> >> To avoid surprises, please test and report. >> >> Thanks, >> Robert >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Sun Mar 20 01:59:48 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 19 Mar 2016 22:59:48 -0700 Subject: [Cython] Cython alpha 0.24 In-Reply-To: References: Message-ID: On Sat, Mar 19, 2016 at 7:51 AM, Jason Madden wrote: > [Ack, really sorry for the premature send. Here's the whole thing:] > > gevent compiles and passes all its tests with this release. > > However, compilation produces 420 warnings of the form: > > warning: gevent/corecext.pyx:1893:4: 'property rpid:' syntax is > deprecated, use '@property' > > That's a lot of warnings. > > There's also this comment in the code, which seems to indicate that the > warnings aren't actually desired, but the test for it includes `if True:` so > warnings are always emitted. I'm a bit confused as to the intent; are the > warnings going to be in the final release? > > def visit_PropertyNode(self, node): > # Suppress warning for our code until we can convert all our uses > over. > if isinstance(node.pos[0], str) or True: > warning(node.pos, "'property %s:' syntax is deprecated, use > '@property'" % node.name, 2) > > As I understand it, this was added with pull #462 which is only in > master/0.24, so there's no way for me to fix these warnings while still > being able to compile on 0.23; is that right? (That is, @property doesn't > work right in 0.23) If so, this would seem to at least merit a mention in > CHANGES.rst? Good point. I've lowered the warning to invisible-by-default for now and added a note in CHANGES.rst. > On Sat, Mar 19, 2016 at 9:44 AM, Jason Madden > wrote: >> >> gevent compiles and passes all its tests with this release. >> >> However, compilation produces 420 warnings of the form: >> >> warning: gevent/corecext.pyx:1893:4: 'property rpid:' syntax is >> deprecated, use '@property' >> >> That's a lot of warnings. >> >> As I understand it, this was added with pull #462 which is only in >> master/0.24, so there >> >> >> On Fri, Mar 11, 2016 at 2:23 AM, Robert Bradshaw >> wrote: >>> >>> We are getting ready to push a new major release. A first alpha is up >>> at http://cython.org/release/Cython-0.24a0.tar.gz >>> >>> To avoid surprises, please test and report. >>> >>> Thanks, >>> Robert >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> https://mail.python.org/mailman/listinfo/cython-devel >> >> > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Mon Mar 21 15:45:17 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 21 Mar 2016 20:45:17 +0100 Subject: [Cython] PEP 498: literal string interpolation (f-strings) Message-ID: <56F04F4D.4070604@behnel.de> Hi all! Jelle Zijlstra has implemented this and contributed his implementation; I've given it some final touches. https://github.com/cython/cython/pull/501 The PEP has some examples for this new (Py3.6) feature: https://www.python.org/dev/peps/pep-0498/ Not sure if this still qualifies for 0.24 at this point, but it certainly is a very nice language feature. I got all CPython tests working (with minor deviations in Py2.6 where str.format() is "less optimal"), so this seems like a reasonable thing to add. Stefan From insertinterestingnamehere at gmail.com Mon Mar 21 22:06:17 2016 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Tue, 22 Mar 2016 02:06:17 +0000 Subject: [Cython] [cython-users] Re: Cython alpha 0.24 In-Reply-To: References: Message-ID: On Mon, Mar 21, 2016 at 6:29 PM Martin Bammer wrote: > Hi, > > I just saw that PyrexTypes.py seems to have a typo in lines 3740 and 3742: > > def __cmp__(self, other): > if isinstance(other, TemplatePlaceholderType): > return cmp(self.name, other.name) > else: > return cmp(type(self), type(other)) > > > PyCharm is complaining that "cmp" is an unresolved reference. > > Regards, > > Martin > > AFAICT, that whole function isn't really necessary. It defines the behavior for when you call `cmp` on an instance of that particular class. In Python 3 that particular builtin function doesn't exist. It'll work fine in Python 2, but since the codebase supports both it's not clear when that would actually be useful. Best, -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at vorpus.org Tue Mar 22 02:45:50 2016 From: njs at vorpus.org (Nathaniel Smith) Date: Mon, 21 Mar 2016 23:45:50 -0700 Subject: [Cython] [ANN] Python compilers workshop at SciPy this year Message-ID: Hi all, I wanted to announce a workshop I'm organizing at SciPy this year, and invite you to attend! What: A two-day workshop bringing together folks working on JIT/AOT compilation in Python. When/where: July 11-12, in Austin, Texas. (This is co-located with SciPy 2016, at the same time as the tutorial sessions, just before the conference proper.) Website: https://python-compilers-workshop.github.io/ Note that I anticipate that we'll be able to get sponsorship funding to cover travel costs for folks who can't get their employers to foot the bill. Cheers, -n -- Nathaniel J. Smith -- https://vorpus.org From matthew.brett at gmail.com Fri Mar 25 14:46:16 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 25 Mar 2016 11:46:16 -0700 Subject: [Cython] Manylinux wheels for Cython In-Reply-To: References: Message-ID: Hi, On Tue, Mar 15, 2016 at 2:58 PM, Matthew Brett wrote: > Hi, > > On Mon, Mar 7, 2016 at 5:47 PM, Matthew Brett wrote: >> Hi, >> >> I don't know whether y'all have been following over at distutils-sig, >> but there's a new distutils PEP that defines a `manylinux` format for >> Linux wheels that work on many different x86 Linux distributions: >> >> https://www.python.org/dev/peps/pep-0513/ >> https://github.com/pypa/manylinux >> >> The latest version of pip will install these, if the client Linux >> system is compatible with the manylinux spec: >> >> https://pip.pypa.io/en/stable/news/ >> >> I've already built and used manylinux Cython wheels, which y'all are >> welcome to test with: >> >> pip install -f https://nipy.bic.berkeley.edu/manylinux cython >> >> (The wheels there don't have the right manylinux filenames yet, but >> they have the same contents as the ones that would go up to pypi). >> >> I've already had good use from these wheels in speeding up project >> builds into docker containers and virtualenvs, and I'd love to upload >> these to pypi. I have permissions on pypi to do this, but I wanted >> to check in with y'all first... > > There is now a test wheel for Cython 0.23.4 and Python 3.5 on the > testpypi server. > > This is me downloading and installing - a matter of a few seconds: > > $ python -m pip install -U pip > Downloading/unpacking pip from > https://pypi.python.org/packages/py2.py3/p/pip/pip-8.1.0-py2.py3-none-any.whl#md5=c6eca6736b2b8f7280fb25e44be7c51b > Downloading pip-8.1.0-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded > Installing collected packages: pip > Found existing installation: pip 1.5.6 > Uninstalling pip: > Successfully uninstalled pip > Successfully installed pip > Cleaning up... > $ pip install -i https://testpypi.python.org/pypi/ cython > Collecting cython > Using cached https://testpypi.python.org/packages/cp35/C/Cython/Cython-0.23.4-cp35-cp35m-manylinux1_x86_64.whl > Installing collected packages: cython > Successfully installed cython-0.23.4 > $ cython --version > Cython version 0.23.4 > > The installed Cython version compiles all the Demo *.pyx files OK. > > See also : https://mail.python.org/pipermail/wheel-builders/2016-March/000050.html A manylinux wheel (gevent) is already the current most-downloaded binary wheel from pypi. A reminder that y'all can test the Cython wheels with: python -m pip install --upgrade pip # You need latest pip pip install -f https://nipy.bic.berkeley.edu/manylinux cython If I don't hear any objections, I plan to upload the Cython manylinux wheels on Monday 28th. Cheers, Matthew From stefan_ml at behnel.de Sat Mar 26 08:47:46 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 26 Mar 2016 13:47:46 +0100 Subject: [Cython] Cython 0.23.5 released Message-ID: <56F684F2.5040701@behnel.de> Hi all! I just pushed a pure bug-fix-only release for the current 0.23 series. Any major changes have been held back for the upcoming 0.24 feature release, so the update is recommended for those who can't wait. https://pypi.python.org/pypi/Cython/0.23.5 Complete changelog follows below. You can get the signed release from here: http://cython.org/release/Cython-0.23.5.tar.gz http://cython.org/release/Cython-0.23.5.zip SHA1 sums: ee455e4f6bedb8d02325da1d4810ab906c3ddc1d Cython-0.23.5.tar.gz a7ced83eab3bd4122513f3ea7d9f50ba1804a400 Cython-0.23.5.zip Have fun, Stefan 0.23.5 (2016-03-26) =================== Bugs fixed ---------- * Compile errors and warnings in integer type conversion code. This fixes ticket 877. Patches by Christian Neukirchen, Nikolaus Rath, Ian Henriksen. * Reference leak when "*args" argument was reassigned in closures. * Truth-testing Unicode strings could waste time and memory in Py3.3+. * Return values of async functions could be ignored and replaced by ``None``. * Compiler crash in CPython 3.6. * Fix prange() to behave identically to range(). The end condition was miscalculated when the range was not exactly divisible by the step. * Optimised ``all(genexpr)``/``any(genexpr)`` calls could warn about unused code. This fixes ticket 876. From jdemeyer at cage.ugent.be Sat Mar 26 09:56:43 2016 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Sat, 26 Mar 2016 14:56:43 +0100 Subject: [Cython] Cython 0.23.5 released In-Reply-To: <56F684F2.5040701@behnel.de> References: <56F684F2.5040701@behnel.de> Message-ID: <56F6951B.5070106@cage.ugent.be> On 2016-03-26 13:47, Stefan Behnel wrote: > I just pushed a pure bug-fix-only release for the current 0.23 series. Any > major changes have been held back for the upcoming 0.24 feature release, so > the update is recommended for those who can't wait. Any idea on the ETA for Cython 0.24? From robertwb at gmail.com Sat Mar 26 12:46:58 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 26 Mar 2016 09:46:58 -0700 Subject: [Cython] Cython 0.23.5 released In-Reply-To: <56F6951B.5070106@cage.ugent.be> References: <56F684F2.5040701@behnel.de> <56F6951B.5070106@cage.ugent.be> Message-ID: On Sat, Mar 26, 2016 at 6:56 AM, Jeroen Demeyer wrote: > On 2016-03-26 13:47, Stefan Behnel wrote: > >> I just pushed a pure bug-fix-only release for the current 0.23 series. Any >> major changes have been held back for the upcoming 0.24 feature release, >> so >> the update is recommended for those who can't wait. >> > > Any idea on the ETA for Cython 0.24? > Likely in the next week. I think I've merged all the major changes that are going in this release, and am cutting a beta now. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Sat Mar 26 13:12:18 2016 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 26 Mar 2016 10:12:18 -0700 Subject: [Cython] Cython 0.24 beta Message-ID: I just pushed a beta for Cython 0.24. It can be downloaded at http://cython.org/release/Cython-0.24b0.tar.gz http://cython.org/release/Cython-0.24b0.zip https://github.com/cython/cython/tree/0.24.x It contains fixes to all the bugs reported with the alpha, as as well as the numerous improvements over 0.23.x. I have pulled in the last major changes I plan to make in this release (but development can continue on master). Thanks for the feedback so far; please continue to build and report. If there are no more issues discovered, this may well be the final release next week. - Robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Mar 28 13:54:21 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 28 Mar 2016 10:54:21 -0700 Subject: [Cython] Manylinux wheels for Cython In-Reply-To: References: Message-ID: Hi, On Fri, Mar 25, 2016 at 11:46 AM, Matthew Brett wrote: > Hi, > > On Tue, Mar 15, 2016 at 2:58 PM, Matthew Brett wrote: >> Hi, >> >> On Mon, Mar 7, 2016 at 5:47 PM, Matthew Brett wrote: >>> Hi, >>> >>> I don't know whether y'all have been following over at distutils-sig, >>> but there's a new distutils PEP that defines a `manylinux` format for >>> Linux wheels that work on many different x86 Linux distributions: >>> >>> https://www.python.org/dev/peps/pep-0513/ >>> https://github.com/pypa/manylinux >>> >>> The latest version of pip will install these, if the client Linux >>> system is compatible with the manylinux spec: >>> >>> https://pip.pypa.io/en/stable/news/ >>> >>> I've already built and used manylinux Cython wheels, which y'all are >>> welcome to test with: >>> >>> pip install -f https://nipy.bic.berkeley.edu/manylinux cython >>> >>> (The wheels there don't have the right manylinux filenames yet, but >>> they have the same contents as the ones that would go up to pypi). >>> >>> I've already had good use from these wheels in speeding up project >>> builds into docker containers and virtualenvs, and I'd love to upload >>> these to pypi. I have permissions on pypi to do this, but I wanted >>> to check in with y'all first... >> >> There is now a test wheel for Cython 0.23.4 and Python 3.5 on the >> testpypi server. >> >> This is me downloading and installing - a matter of a few seconds: >> >> $ python -m pip install -U pip >> Downloading/unpacking pip from >> https://pypi.python.org/packages/py2.py3/p/pip/pip-8.1.0-py2.py3-none-any.whl#md5=c6eca6736b2b8f7280fb25e44be7c51b >> Downloading pip-8.1.0-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded >> Installing collected packages: pip >> Found existing installation: pip 1.5.6 >> Uninstalling pip: >> Successfully uninstalled pip >> Successfully installed pip >> Cleaning up... >> $ pip install -i https://testpypi.python.org/pypi/ cython >> Collecting cython >> Using cached https://testpypi.python.org/packages/cp35/C/Cython/Cython-0.23.4-cp35-cp35m-manylinux1_x86_64.whl >> Installing collected packages: cython >> Successfully installed cython-0.23.4 >> $ cython --version >> Cython version 0.23.4 >> >> The installed Cython version compiles all the Demo *.pyx files OK. >> >> See also : https://mail.python.org/pipermail/wheel-builders/2016-March/000050.html > > A manylinux wheel (gevent) is already the current most-downloaded > binary wheel from pypi. > > A reminder that y'all can test the Cython wheels with: > > python -m pip install --upgrade pip # You need latest pip > pip install -f https://nipy.bic.berkeley.edu/manylinux cython > > If I don't hear any objections, I plan to upload the Cython manylinux > wheels on Monday 28th. I uploaded manylinux wheels for Cython 0.23.5. If you're on Linux, and you upgrade pip to 8.1.1 (current) you should now get Cython via a manylinux wheel by default. Please do test and let me know of any problems. Cheers, Matthew From linkmauve at linkmauve.fr Mon Mar 28 19:04:09 2016 From: linkmauve at linkmauve.fr (Emmanuel Gil Peyrot) Date: Tue, 29 Mar 2016 01:04:09 +0200 Subject: [Cython] Manylinux wheels for Cython In-Reply-To: References: Message-ID: <20160328230409.GA16433@yuyuko> On Mon, Mar 28, 2016 at 10:54:21AM -0700, Matthew Brett wrote: [?] > I uploaded manylinux wheels for Cython 0.23.5. > > If you're on Linux, and you upgrade pip to 8.1.1 (current) you should > now get Cython via a manylinux wheel by default. > > Please do test and let me know of any problems. Thanks, this significantly improves the install-time required for poezio for people who don?t already have cython installed. Do you plan on publishing AArch64 and ARMv7h wheels as well? Those are the systems most impacted by the compile time of cython. > > Cheers, > > Matthew > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel -- Emmanuel Gil Peyrot From njs at vorpus.org Mon Mar 28 20:14:56 2016 From: njs at vorpus.org (Nathaniel Smith) Date: Mon, 28 Mar 2016 17:14:56 -0700 Subject: [Cython] Manylinux wheels for Cython In-Reply-To: <20160328230409.GA16433@yuyuko> References: <20160328230409.GA16433@yuyuko> Message-ID: On Mon, Mar 28, 2016 at 4:04 PM, Emmanuel Gil Peyrot wrote: > On Mon, Mar 28, 2016 at 10:54:21AM -0700, Matthew Brett wrote: > [?] >> I uploaded manylinux wheels for Cython 0.23.5. >> >> If you're on Linux, and you upgrade pip to 8.1.1 (current) you should >> now get Cython via a manylinux wheel by default. >> >> Please do test and let me know of any problems. > > Thanks, this significantly improves the install-time required for > poezio for people who don?t already have cython installed. > > Do you plan on publishing AArch64 and ARMv7h wheels as well? Those are > the systems most impacted by the compile time of cython. PEP 513 only standardizes linux wheels for x86-{32,64}, because its strategy is to say "hey, in practice everyone is careful to maintain backwards ABI compatibility with RHEL-cough-I-mean-CentOS 5, so let's use that as our standard ABI". But CentOS 5 is only available for x86-{32,64}. It would be great if there's some way to standardize an ABI that lets us distribute ARM wheels -- I just don't know enough about ARM to have any idea how to do that :-). One possibility would be to standardize a way to distribute wheels that target particular distributions, so you could have e.g. a wheel for Raspbian specifically -- Nate Coraor was interested in working on this. If you send an email to distutils-sig volunteering to help then you might get some takers :-). Alternatively maybe there is some minimum baseline ABI that most/all ARM distributions actually do agree on in practice, similar to CentOS 5 for x86 variants... that could also be used to standardize something. But either way it would need someone with the appropriate expertise (or willingness to fake it) to step up and make a proposal on distutils-sig. -n -- Nathaniel J. Smith -- https://vorpus.org From matthew.brett at gmail.com Thu Mar 31 01:50:16 2016 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 30 Mar 2016 22:50:16 -0700 Subject: [Cython] [cython-users] Re: Cython 0.23.5 released In-Reply-To: <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> References: <56F684F2.5040701@behnel.de> <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> Message-ID: On Wed, Mar 30, 2016 at 2:35 PM, Forest Gregg wrote: > Thanks! > > Are there also plans to post Windows wheels for cython 0.23.5 to pypi like > there was for 0.23.4? I did that earlier today, using an Appveyor build system : https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml https://ci.appveyor.com/project/matthew-brett/cython-wheels http://win-wheels.scikit-image.org/ Do they work for you? Best, Matthew From jason.madden at nextthought.com Thu Mar 31 09:11:21 2016 From: jason.madden at nextthought.com (Jason Madden) Date: Thu, 31 Mar 2016 08:11:21 -0500 Subject: [Cython] [cython-users] Re: Cython 0.23.5 released In-Reply-To: References: <56F684F2.5040701@behnel.de> <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> Message-ID: <7AA9FD11-47A4-43E7-94B8-8A08B3C77F8D@nextthought.com> > On Mar 31, 2016, at 00:50, Matthew Brett wrote: > > On Wed, Mar 30, 2016 at 2:35 PM, Forest Gregg wrote: >> Thanks! >> >> Are there also plans to post Windows wheels for cython 0.23.5 to pypi like >> there was for 0.23.4? > > I did that earlier today, using an Appveyor build system : > > https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml > https://ci.appveyor.com/project/matthew-brett/cython-wheels > http://win-wheels.scikit-image.org/ > > Do they work for you? They work for building gevent on Appveyor. The cython binary wheels cut gevent's total build time by more than half (they save almost 10 minutes). Thanks! Jason From jason.madden at nextthought.com Thu Mar 31 09:13:46 2016 From: jason.madden at nextthought.com (Jason Madden) Date: Thu, 31 Mar 2016 06:13:46 -0700 (PDT) Subject: [Cython] [cython-users] Re: Cython 0.23.5 released In-Reply-To: References: <56F684F2.5040701@behnel.de> <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> Message-ID: On Thursday, 31 March 2016 00:50:57 UTC-5, Matthew Brett wrote: > > On Wed, Mar 30, 2016 at 2:35 PM, Forest Gregg > wrote: > > Thanks! > > > > Are there also plans to post Windows wheels for cython 0.23.5 to pypi > like > > there was for 0.23.4? > > I did that earlier today, using an Appveyor build system : > > https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml > https://ci.appveyor.com/project/matthew-brett/cython-wheels > http://win-wheels.scikit-image.org/ > > Do they work for you? > They work for building gevent on Appveyor. The cython binary wheels cut gevent's total build time by more than half (they save almost 10 minutes). Thanks! Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Mar 31 09:40:03 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 31 Mar 2016 15:40:03 +0200 Subject: [Cython] [cython-users] Re: Cython 0.23.5 released In-Reply-To: References: <56F684F2.5040701@behnel.de> <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> Message-ID: <56FD28B3.4020007@behnel.de> Jason Madden schrieb am 31.03.2016 um 15:13: > On Thursday, 31 March 2016 00:50:57 UTC-5, Matthew Brett wrote: >> >> On Wed, Mar 30, 2016 at 2:35 PM, Forest Gregg > > wrote: >>> Thanks! >>> >>> Are there also plans to post Windows wheels for cython 0.23.5 to pypi >> like >>> there was for 0.23.4? >> >> I did that earlier today, using an Appveyor build system : >> >> https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml >> https://ci.appveyor.com/project/matthew-brett/cython-wheels >> http://win-wheels.scikit-image.org/ >> >> Do they work for you? > > They work for building gevent on Appveyor. The cython binary wheels cut > gevent's total build time by more than half (they save almost 10 minutes). That makes me wonder if we shouldn't just provide pure Python wheels of Cython, i.e. without compiling any binary extensions in them. For one-time installs, it would only be marginally slower than with a compiled Cython, but it would work on all platforms with all Python versions. I guess the drawback would be that it would be more hassle for people who really want to install Cython permanently for themselves... Is there a way to tag a wheel somehow as "special"? I doubt that Nick&friends thought about the special Cython installation use case when designing the format... Stefan From njs at vorpus.org Thu Mar 31 11:42:39 2016 From: njs at vorpus.org (Nathaniel Smith) Date: Thu, 31 Mar 2016 08:42:39 -0700 Subject: [Cython] [cython-users] Re: Cython 0.23.5 released In-Reply-To: <56FD28B3.4020007@behnel.de> References: <56F684F2.5040701@behnel.de> <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> <56FD28B3.4020007@behnel.de> Message-ID: On Mar 31, 2016 6:40 AM, "Stefan Behnel" wrote: > > Jason Madden schrieb am 31.03.2016 um 15:13: > > On Thursday, 31 March 2016 00:50:57 UTC-5, Matthew Brett wrote: > >> > >> On Wed, Mar 30, 2016 at 2:35 PM, Forest Gregg >> > wrote: > >>> Thanks! > >>> > >>> Are there also plans to post Windows wheels for cython 0.23.5 to pypi > >> like > >>> there was for 0.23.4? > >> > >> I did that earlier today, using an Appveyor build system : > >> > >> https://github.com/MacPython/cython-wheels/blob/master/appveyor.yml > >> https://ci.appveyor.com/project/matthew-brett/cython-wheels > >> http://win-wheels.scikit-image.org/ > >> > >> Do they work for you? > > > > They work for building gevent on Appveyor. The cython binary wheels cut > > gevent's total build time by more than half (they save almost 10 minutes). > > That makes me wonder if we shouldn't just provide pure Python wheels of > Cython, i.e. without compiling any binary extensions in them. For one-time > installs, it would only be marginally slower than with a compiled Cython, > but it would work on all platforms with all Python versions. > > I guess the drawback would be that it would be more hassle for people who > really want to install Cython permanently for themselves... > > Is there a way to tag a wheel somehow as "special"? I doubt that > Nick&friends thought about the special Cython installation use case when > designing the format... The general case is "library with optional native acceleration". I think one thing is that if there's both a generic and a specific wheel then pip will prefer the specific one? So an arch-specific wheel will get higher priority than a py2.py3-none-any wheel. Of course this only works if both are available -- it's more of a fallback thing. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Mar 31 14:28:50 2016 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 31 Mar 2016 20:28:50 +0200 Subject: [Cython] [cython-users] Re: Cython 0.23.5 released In-Reply-To: References: <56F684F2.5040701@behnel.de> <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> <56FD28B3.4020007@behnel.de> Message-ID: <56FD6C62.3070707@behnel.de> Nathaniel Smith schrieb am 31.03.2016 um 17:42: > On Mar 31, 2016 6:40 AM, "Stefan Behnel" wrote: >> That makes me wonder if we shouldn't just provide pure Python wheels of >> Cython, i.e. without compiling any binary extensions in them. For one-time >> installs, it would only be marginally slower than with a compiled Cython, >> but it would work on all platforms with all Python versions. >> >> I guess the drawback would be that it would be more hassle for people who >> really want to install Cython permanently for themselves... >> >> Is there a way to tag a wheel somehow as "special"? I doubt that >> Nick&friends thought about the special Cython installation use case when >> designing the format... > > The general case is "library with optional native acceleration". I think > one thing is that if there's both a generic and a specific wheel then pip > will prefer the specific one? So an arch-specific wheel will get higher > priority than a py2.py3-none-any wheel. Of course this only works if both > are available -- it's more of a fallback thing. We could provide a separate release version for the wheel, e.g. 0.24 for normal sdist and binary builds and 0.24py for the unaccelerated pure Python wheel. People could then "pip install Cython==0.24py" on their CI servers and get a fast wheel installation. The wheel is really easy to build: pip install wheel python setup.py bdist_wheel --universal --no-cython-compile This quickly creates a file dist/Cython-0.24b0-py2.py3-none-any.whl Py3.2 wouldn't be supported, but that's it. Since Cython shouldn't normally show up in any requirements.txt files itself, and thus would usually be installed separately in CI builds, this scheme should be easy to apply for users. Using different versions is better than having different package names because the later would quickly lead to confusion when both packages get installed by accident and try to unpack the same files. We need to take care that PyPI's version sorting doesn't get confused by this, though. "pip install Cython" should still install the latest *accelerated* version. Does anyone know if the trove classifier Development Status :: 7 - Inactive has any effect on pip? Or should we mark the wheel package as "3 - Alpha" release? That should prevent pip from installing it by default (IIRC). Stefan From njs at vorpus.org Thu Mar 31 15:23:56 2016 From: njs at vorpus.org (Nathaniel Smith) Date: Thu, 31 Mar 2016 12:23:56 -0700 Subject: [Cython] [cython-users] Re: Cython 0.23.5 released In-Reply-To: <56FD6C62.3070707@behnel.de> References: <56F684F2.5040701@behnel.de> <5528aade-00d1-41df-95db-5d65af22aa3c@googlegroups.com> <56FD28B3.4020007@behnel.de> <56FD6C62.3070707@behnel.de> Message-ID: Pip ignores trove classifiers entirely. You could overload PEP 440 prerelease versions, which pip ignores by default, but I'm not sure I'd advise it, since those are rather useful for distributing prereleases :-). I think I missed something in the initial motivation here: if there's a precompiled binary wheel available for some platform, then that's what CI servers should use, right? So the goal here is to provide some convenient way for folks on unusual platforms without wheels -- like fbsd or arm Linux -- to get the behavior that "pip install Cython" does the lengthy build-from-source thing, but on CI servers they can instead run "pip install [something]" to get a quick install? And what's wrong with the current "pip install Cython --install-option="--no-cython-compile"? On Mar 31, 2016 11:29 AM, "Stefan Behnel" wrote: > Nathaniel Smith schrieb am 31.03.2016 um 17:42: > > On Mar 31, 2016 6:40 AM, "Stefan Behnel" wrote: > >> That makes me wonder if we shouldn't just provide pure Python wheels of > >> Cython, i.e. without compiling any binary extensions in them. For > one-time > >> installs, it would only be marginally slower than with a compiled > Cython, > >> but it would work on all platforms with all Python versions. > >> > >> I guess the drawback would be that it would be more hassle for people > who > >> really want to install Cython permanently for themselves... > >> > >> Is there a way to tag a wheel somehow as "special"? I doubt that > >> Nick&friends thought about the special Cython installation use case when > >> designing the format... > > > > The general case is "library with optional native acceleration". I think > > one thing is that if there's both a generic and a specific wheel then pip > > will prefer the specific one? So an arch-specific wheel will get higher > > priority than a py2.py3-none-any wheel. Of course this only works if both > > are available -- it's more of a fallback thing. > > We could provide a separate release version for the wheel, e.g. 0.24 for > normal sdist and binary builds and 0.24py for the unaccelerated pure Python > wheel. People could then "pip install Cython==0.24py" on their CI servers > and get a fast wheel installation. > > The wheel is really easy to build: > > pip install wheel > python setup.py bdist_wheel --universal --no-cython-compile > > This quickly creates a file > > dist/Cython-0.24b0-py2.py3-none-any.whl > > Py3.2 wouldn't be supported, but that's it. > > Since Cython shouldn't normally show up in any requirements.txt files > itself, and thus would usually be installed separately in CI builds, this > scheme should be easy to apply for users. > > Using different versions is better than having different package names > because the later would quickly lead to confusion when both packages get > installed by accident and try to unpack the same files. We need to take > care that PyPI's version sorting doesn't get confused by this, though. "pip > install Cython" should still install the latest *accelerated* version. > > Does anyone know if the trove classifier > > Development Status :: 7 - Inactive > > has any effect on pip? Or should we mark the wheel package as "3 - Alpha" > release? That should prevent pip from installing it by default (IIRC). > > Stefan > > -- > > --- > You received this message because you are subscribed to the Google Groups > "cython-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to cython-users+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: