From insertinterestingnamehere at gmail.com Thu Oct 1 07:31:49 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Thu, 01 Oct 2015 05:31:49 +0000 Subject: [Cython] Non-type template parameters In-Reply-To: <560B947E.1020207@behnel.de> References: <560B947E.1020207@behnel.de> Message-ID: > > I'd like to see this feature merged rather sooner than later. Would it help > if we only require a type to be there (so that people get the syntax right > in their code) but do not validate it for now? > > Stefan > > Stefan, Sorry for the delay, I've been mulling over what to do about this and haven't been fully satisfied with any of the answers. Not requiring the checks for now would definitely make this faster to implement though. My biggest concern is that, in C++, the types can actually play a significant role in the implementation of a given operation, and many of the things that go into determining what type may be returned are things that we don't want to have visible in Cython. Features like SFINAE and enable_if may be difficult for us to support, but syntactically, they don't use much fancy syntax. At some point, it seems to be in the best interest of our users if we let the C++ compiler finish off the type checking rather than requiring that all the type information be present in Cython. As near as I can tell, adding type specifiers to non-type template parameters won't put us into much murky water though, so I'll take another shot at adding the new syntax. Thanks for following up here. Best, -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Fri Oct 2 12:11:47 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Oct 2015 13:11:47 +0300 Subject: [Cython] Python 3 wheels including bytecode files. Message-ID: After getting AppVeyor to work, I noticed that Python 3 wheels do include __pycache__ directory. This also happens with in OS X as well as Linux. I think the root of the issue resides in setup.py, in the build_ext command class workaround that seems to be a legacy thing from the time Cython required 2to3. BTW, in setup.py, compile_cython_modules() follows quite a different path in Python 2 versus 3. Unless some of you believe we should keep things that way, I'll try to fix setup.py to be as much independent as possible of the major Python version. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Fri Oct 2 13:26:33 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 2 Oct 2015 13:26:33 +0200 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: References: Message-ID: <560E69E9.6050609@behnel.de> Lisandro Dalcin schrieb am 02.10.2015 um 12:11: > After getting AppVeyor to work, I noticed that Python 3 wheels do > include __pycache__ directory. This also happens with in OS X as well > as Linux. > > I think the root of the issue resides in setup.py, in the build_ext > command class workaround that seems to be a legacy thing from the time > Cython required 2to3. BTW, in setup.py, compile_cython_modules() > follows quite a different path in Python 2 versus 3. I already applied a change in master, would be worth seeing if that has an effect here, too. There is certainly more that can be done. As you said, this is legacy code (of a not so old legacy, though). > Unless some of you believe we should keep things that way, I'll try to > fix setup.py to be as much independent as possible of the major Python > version. Please do. :) Stefan From stefan_ml at behnel.de Fri Oct 2 13:30:14 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 2 Oct 2015 13:30:14 +0200 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: <560E69E9.6050609@behnel.de> References: <560E69E9.6050609@behnel.de> Message-ID: <560E6AC6.6040400@behnel.de> Stefan Behnel schrieb am 02.10.2015 um 13:26: > Lisandro Dalcin schrieb am 02.10.2015 um 12:11: >> After getting AppVeyor to work, I noticed that Python 3 wheels do >> include __pycache__ directory. This also happens with in OS X as well >> as Linux. >> >> I think the root of the issue resides in setup.py, in the build_ext >> command class workaround that seems to be a legacy thing from the time >> Cython required 2to3. BTW, in setup.py, compile_cython_modules() >> follows quite a different path in Python 2 versus 3. > > I already applied a change in master, would be worth seeing if that has an > effect here, too. There is certainly more that can be done. As you said, > this is legacy code (of a not so old legacy, though). > > >> Unless some of you believe we should keep things that way, I'll try to >> fix setup.py to be as much independent as possible of the major Python >> version. Ah - note that Python 3.2 still uses the unicode fixer of 2to3, so the code is not entirely legacy. Would be nice if it could be moved out of the way, though. Especially since it will eventually disappear entirely. Stefan From dalcinl at gmail.com Fri Oct 2 14:07:10 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Oct 2015 15:07:10 +0300 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: <560E6AC6.6040400@behnel.de> References: <560E69E9.6050609@behnel.de> <560E6AC6.6040400@behnel.de> Message-ID: On 2 October 2015 at 14:30, Stefan Behnel wrote: > Ah - note that Python 3.2 still uses the unicode fixer of 2to3, so the code > is not entirely legacy. Would be nice if it could be moved out of the way, > though. Especially since it will eventually disappear entirely. Yes, I'm aware of this. I'll push a fix in branch 0.23.x (I'm doing my tests in OS X), then see what happens with the AppVeyor builds. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From dalcinl at gmail.com Fri Oct 2 15:18:38 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Oct 2015 16:18:38 +0300 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: References: <560E69E9.6050609@behnel.de> <560E6AC6.6040400@behnel.de> Message-ID: On 2 October 2015 at 15:07, Lisandro Dalcin wrote: > On 2 October 2015 at 14:30, Stefan Behnel wrote: >> Ah - note that Python 3.2 still uses the unicode fixer of 2to3, so the code >> is not entirely legacy. Would be nice if it could be moved out of the way, >> though. Especially since it will eventually disappear entirely. > > Yes, I'm aware of this. I'll push a fix in branch 0.23.x (I'm doing my > tests in OS X), then see what happens with the AppVeyor builds. > AppVeyor builds succeeded: https://ci.appveyor.com/project/cython/cython/build/0.23.x-4 The Python 3 wheels no longer have *.pyc files, so I'm confident this is fixed for other platforms as well. Stefan, could you manage to merge branch 0.23.x into master? PS: At some point, we should try to really run the testsuite in AppVeyor. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Fri Oct 2 16:01:41 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 2 Oct 2015 16:01:41 +0200 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: References: <560E69E9.6050609@behnel.de> <560E6AC6.6040400@behnel.de> Message-ID: <560E8E45.5010602@behnel.de> Lisandro Dalcin schrieb am 02.10.2015 um 15:18: > On 2 October 2015 at 15:07, Lisandro Dalcin wrote: >> On 2 October 2015 at 14:30, Stefan Behnel wrote: >>> Ah - note that Python 3.2 still uses the unicode fixer of 2to3, so the code >>> is not entirely legacy. Would be nice if it could be moved out of the way, >>> though. Especially since it will eventually disappear entirely. >> >> Yes, I'm aware of this. I'll push a fix in branch 0.23.x (I'm doing my >> tests in OS X), then see what happens with the AppVeyor builds. > > AppVeyor builds succeeded: > https://ci.appveyor.com/project/cython/cython/build/0.23.x-4 > > The Python 3 wheels no longer have *.pyc files, so I'm confident this > is fixed for other platforms as well. > > Stefan, could you manage to merge branch 0.23.x into master? Done, thanks! github just told me that my merge landed as commit 11111. :) I also changed a couple of minor things: https://github.com/cython/cython/commit/64da2480e88d8cd3eec2b9a5d2922307623c3cd5 I guess the # XXX hack for '*.pyx ' sources referred to the setuptools quirk that stubbornly replaces .pyx by .c? > PS: At some point, we should try to really run the testsuite in AppVeyor. What's not working there? Stefan From dalcinl at gmail.com Fri Oct 2 16:24:56 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Oct 2015 17:24:56 +0300 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: <560E8E45.5010602@behnel.de> References: <560E69E9.6050609@behnel.de> <560E6AC6.6040400@behnel.de> <560E8E45.5010602@behnel.de> Message-ID: On 2 October 2015 at 17:01, Stefan Behnel wrote: > Lisandro Dalcin schrieb am 02.10.2015 um 15:18: >> On 2 October 2015 at 15:07, Lisandro Dalcin wrote: >>> On 2 October 2015 at 14:30, Stefan Behnel wrote: >>>> Ah - note that Python 3.2 still uses the unicode fixer of 2to3, so the code >>>> is not entirely legacy. Would be nice if it could be moved out of the way, >>>> though. Especially since it will eventually disappear entirely. >>> >>> Yes, I'm aware of this. I'll push a fix in branch 0.23.x (I'm doing my >>> tests in OS X), then see what happens with the AppVeyor builds. >> >> AppVeyor builds succeeded: >> https://ci.appveyor.com/project/cython/cython/build/0.23.x-4 >> >> The Python 3 wheels no longer have *.pyc files, so I'm confident this >> is fixed for other platforms as well. >> >> Stefan, could you manage to merge branch 0.23.x into master? > > Done, thanks! github just told me that my merge landed as commit 11111. :) > > I also changed a couple of minor things: > > https://github.com/cython/cython/commit/64da2480e88d8cd3eec2b9a5d2922307623c3cd5 > I don't like your change. Before, the self-cythonization was triggeres when you actually needed to compile the extensions, now a simple "setup.py --version" will do. Try yourself: $ git clean -dxf $ python setup.py --version > I guess the > > # XXX hack for '*.pyx ' sources > > referred to the setuptools quirk that stubbornly replaces .pyx by .c? > Yes, indeed. > >> PS: At some point, we should try to really run the testsuite in AppVeyor. > > What's not working there? > Right now, the "test" step in AppVeyor invokes "python setup.py test", which basically does nothing. I mean, you should run "python runtests.py" intead. https://github.com/cython/cython/blob/0.23.x/appveyor.yml#L73 -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Sat Oct 3 14:30:22 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 3 Oct 2015 14:30:22 +0200 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: References: <560E69E9.6050609@behnel.de> <560E6AC6.6040400@behnel.de> <560E8E45.5010602@behnel.de> Message-ID: <560FCA5E.8060007@behnel.de> Lisandro Dalcin schrieb am 02.10.2015 um 16:24: > On 2 October 2015 at 17:01, Stefan Behnel wrote: >> Lisandro Dalcin schrieb am 02.10.2015 um 15:18: >>> On 2 October 2015 at 15:07, Lisandro Dalcin wrote: >>>> On 2 October 2015 at 14:30, Stefan Behnel wrote: >>>>> Ah - note that Python 3.2 still uses the unicode fixer of 2to3, so the code >>>>> is not entirely legacy. Would be nice if it could be moved out of the way, >>>>> though. Especially since it will eventually disappear entirely. >>>> >>>> Yes, I'm aware of this. I'll push a fix in branch 0.23.x (I'm doing my >>>> tests in OS X), then see what happens with the AppVeyor builds. >>> >>> AppVeyor builds succeeded: >>> https://ci.appveyor.com/project/cython/cython/build/0.23.x-4 >>> >>> The Python 3 wheels no longer have *.pyc files, so I'm confident this >>> is fixed for other platforms as well. >>> >>> Stefan, could you manage to merge branch 0.23.x into master? >> >> Done, thanks! github just told me that my merge landed as commit 11111. :) >> >> I also changed a couple of minor things: >> >> https://github.com/cython/cython/commit/64da2480e88d8cd3eec2b9a5d2922307623c3cd5 > > I don't like your change. Before, the self-cythonization was triggeres > when you actually needed to compile the extensions, now a simple > "setup.py --version" will do. Thanks, hadn't thought of that. https://github.com/cython/cython/commit/0a2e03f9f41fb405055afd7aa0bb57286293a3f6 >>> PS: At some point, we should try to really run the testsuite in AppVeyor. > Right now, the "test" step in AppVeyor invokes "python setup.py test", > which basically does nothing. I mean, you should run "python > runtests.py" intead. > https://github.com/cython/cython/blob/0.23.x/appveyor.yml#L73 Let's try it in master first. https://github.com/cython/cython/commit/bc88b4ae02510d68014839769a1ed47faa7af077 Stefan From josh at ayers.pw Sat Oct 3 18:04:10 2015 From: josh at ayers.pw (Josh Ayers) Date: Sat, 03 Oct 2015 09:04:10 -0700 Subject: [Cython] Cython 0.23.3 compiler warning with fused types Message-ID: <1443888250.1590436.400376897.1BA45B1E@webmail.messagingengine.com> Cython devs, Using Cython 0.23.3 and Python 3.4.3, the following Cython code: ctypedef fused floating: float double cpdef add_one(floating [:] array): cdef Py_ssize_t i for i in range(array.shape[0]): array[i] += 1.0 produces the C code below. It raises a compiler warning because a long is assigned to a char. char __pyx_v_kind; ... long __pyx_t_11; ... __pyx_t_11 = __Pyx_PyObject_Ord(__pyx_t_8); ... __pyx_v_kind = __pyx_t_11; Thanks, Josh Ayers From jdemeyer at cage.ugent.be Mon Oct 5 16:04:36 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Mon, 05 Oct 2015 16:04:36 +0200 Subject: [Cython] __dealloc__ called even if __cinit__ failed Message-ID: <56128374.5010703@cage.ugent.be> Hello, I'm not entirely sure if this is a feature or bug... I noticed that, if a __cinit__ call raises an exception on an object, that __dealloc__ on that same object is still called. The problem is that __cinit__ is supposed to set up the C-level attributes for that object. If this somehow fails, then the object isn't in a consistent state yet, so it makes no sense to call __dealloc__ on it. Do you consider this a Cython bug or should I manually protect my code against this? Thanks, Jeroen. From robertwb at gmail.com Mon Oct 5 16:57:18 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 5 Oct 2015 07:57:18 -0700 Subject: [Cython] __dealloc__ called even if __cinit__ failed In-Reply-To: <56128374.5010703@cage.ugent.be> References: <56128374.5010703@cage.ugent.be> Message-ID: If cinit fails half way through you still may need to do some cleanup, hence dealloc which must be aware of this. On Oct 5, 2015 7:12 AM, "Jeroen Demeyer" wrote: > Hello, > > I'm not entirely sure if this is a feature or bug... > > I noticed that, if a __cinit__ call raises an exception on an object, that > __dealloc__ on that same object is still called. > > The problem is that __cinit__ is supposed to set up the C-level attributes > for that object. If this somehow fails, then the object isn't in a > consistent state yet, so it makes no sense to call __dealloc__ on it. > > Do you consider this a Cython bug or should I manually protect my code > against this? > > > Thanks, > Jeroen. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Mon Oct 5 20:02:15 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 5 Oct 2015 21:02:15 +0300 Subject: [Cython] Python 3 wheels including bytecode files. In-Reply-To: <560FCA5E.8060007@behnel.de> References: <560E69E9.6050609@behnel.de> <560E6AC6.6040400@behnel.de> <560E8E45.5010602@behnel.de> <560FCA5E.8060007@behnel.de> Message-ID: On 3 October 2015 at 15:30, Stefan Behnel wrote: > Let's try it in master first. > > https://github.com/cython/cython/commit/bc88b4ae02510d68014839769a1ed47faa7af077 Stefan, I'm starting to fix issues here and there. I have no idea how to fix some tests, in such cases I'm just dissabling them at the command line. At some point, I'll need help to resolve them. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From greg.ewing at canterbury.ac.nz Tue Oct 6 06:31:56 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Oct 2015 17:31:56 +1300 Subject: [Cython] __dealloc__ called even if __cinit__ failed In-Reply-To: <56128374.5010703@cage.ugent.be> References: <56128374.5010703@cage.ugent.be> Message-ID: <56134EBC.3020309@canterbury.ac.nz> Jeroen Demeyer wrote: > The problem is that __cinit__ is supposed to set up the C-level > attributes for that object. If this somehow fails, then the object isn't > in a consistent state yet, so it makes no sense to call __dealloc__ on it. > > Do you consider this a Cython bug or should I manually protect my code > against this? It's probably a good thing that __dealloc__ is called, because it provides an opportunity to clean up a partially initialised object that would otherwise leak memory. It does mean that you need to design your __dealloc__ to be robust against this situation. However, the alternative would be to catch exceptions in __cinit__ and clean up there, which means you have cleanup code in two places. -- Greg From atash at google.com Thu Oct 8 01:06:01 2015 From: atash at google.com (Masood Malekghassemi) Date: Wed, 7 Oct 2015 16:06:01 -0700 Subject: [Cython] Cython coverage and multiple projects Message-ID: I'd asked this of Robert Bradshaw earlier and was pointed in this direction (specifically towards Stefan), so here goes! I'm having trouble with getting Cython to play nicely with the coverage.py tool via the Cython.Coverage plug-in. We have tests that are running from a different setup.py project than the one which contains the Cython files (the projects respectively being for the duration of this discussion the 'tests' project and the 'source' project). The generated coverage file contains the line data for individual .pyx files, but the paths of the .pyx files themselves *within* the .coverage file appear to be rooted in the tests project rather than the site-packages folder in which the source project's files were installed (i.e. paths that never exist throughout the edit-build-test cycle). After editing the .coverage file manually to point to the source project directory, coverage.py reports that the .pyx files are not python files and errors out... but the .coveragerc file contains the plug-in module via `[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython coverage.py plug-in to catch those files and handle them instead of letting the rest of coverage.py get confused over them. Furthermore, this is against the 4.0 coverage.py release rather than the 4.0b2 coverage.py release (which appears to be the version against which the latest edit to the Cython code base was made). I don't see any API differences between the two in the plug-ins, but I might have missed something so maybe that's relevant. A secondary issue: the .pyx files are never copied over to the installation directory, so even if the paths were 'correct' (with respect to being copied into the site-packages folder or being accessible via developer mode pip-installs), they wouldn't be found. This seems like it wouldn't necessarily be an issue from a cursory glance at the source code as the file tracer in the Cython coverage.py plug-in appears to spit out blank lines in such a scenario. Complicating matters further, we're running the tests through pytest. That said, I've given up on trying to get pytest-cov to play nicely with this until I can get a manual invocation of `coverage report ...` to produce output describing the .pyx files. Thanks for reading this far. Got any pointers? ---------------------------------------------------------------- Context: https://github.com/soltanmm/grpc/tree/cy-wip/src/python Basic steps to run the tests: - cd $GRPC_REPOSITORY_ROOT - make - CONFIG=opt ./tools/run_tests/build_python.sh 2.7 - CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh -------------- next part -------------- An HTML attachment was scrubbed... URL: From lrq3000 at gmail.com Thu Oct 8 16:35:34 2015 From: lrq3000 at gmail.com (Stephen LARROQUE) Date: Thu, 8 Oct 2015 16:35:34 +0200 Subject: [Cython] cython-devel Digest, Vol 57, Issue 5 In-Reply-To: References: Message-ID: Hello Masood, I can only answer the second issue. You need to explicitly include the *.pyx files in your MANIFEST.in, because else this .pyx files aren't automatically managed by the distutils package so it doesn't know it has to add the .pyx files. I have recently packaged two libraries, with different hierarchy layouts, so you can maybe use one of them as an example to fix your packaging: First project with a usual package structure: https://github.com/lrq3000/unireedsolomon/blob/master/MANIFEST.in https://github.com/lrq3000/unireedsolomon/blob/master/setup.py Second project with a flatout structure: https://github.com/lrq3000/reedsolomon/blob/master/MANIFEST.in https://github.com/lrq3000/reedsolomon/blob/master/setup.py The commands I use to locally build the package are: python setup.py sdist --formats=gztar,zip bdist_wininst --plat-name=win32 python setup.py sdist bdist_egg bdist_wheel --plat-name=win32 If your packaging config is correct, you should see the .pyx files in those builds. And to build all other formats and upload to pypi, I use the python library Twine: twine upload dist/* Hope this can help you. Best regards, Stephen Larroque 2015-10-08 12:00 GMT+02:00 : > Send cython-devel mailing list submissions to > cython-devel at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/cython-devel > or, via email, send a message with subject or body 'help' to > cython-devel-request at python.org > > You can reach the person managing the list at > cython-devel-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cython-devel digest..." > > > Today's Topics: > > 1. Cython coverage and multiple projects (Masood Malekghassemi) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 7 Oct 2015 16:06:01 -0700 > From: Masood Malekghassemi > To: cython-devel at python.org > Subject: [Cython] Cython coverage and multiple projects > Message-ID: > g8w at mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I'd asked this of Robert Bradshaw earlier and was pointed in this direction > (specifically towards Stefan), so here goes! > > I'm having trouble with getting Cython to play nicely with the coverage.py > tool via the Cython.Coverage plug-in. > > We have tests that are running from a different setup.py project than the > one which contains the Cython files (the projects respectively being for > the duration of this discussion the 'tests' project and the 'source' > project). The generated coverage file contains the line data for individual > .pyx files, but the paths of the .pyx files themselves *within* the > .coverage file appear to be rooted in the tests project rather than the > site-packages folder in which the source project's files were installed > (i.e. paths that never exist throughout the edit-build-test cycle). > > After editing the .coverage file manually to point to the source project > directory, coverage.py reports that the .pyx files are not python files and > errors out... but the .coveragerc file contains the plug-in module via > `[run]\n plugins = Cython.Coverage`, so I'd have expected the Cython > coverage.py plug-in to catch those files and handle them instead of letting > the rest of coverage.py get confused over them. > > Furthermore, this is against the 4.0 coverage.py release rather than the > 4.0b2 coverage.py release (which appears to be the version against which > the latest edit to the Cython code base was made). I don't see any API > differences between the two in the plug-ins, but I might have missed > something so maybe that's relevant. > > A secondary issue: the .pyx files are never copied over to the installation > directory, so even if the paths were 'correct' (with respect to being > copied into the site-packages folder or being accessible via developer mode > pip-installs), they wouldn't be found. This seems like it wouldn't > necessarily be an issue from a cursory glance at the source code as the > file tracer in the Cython coverage.py plug-in appears to spit out blank > lines in such a scenario. > > Complicating matters further, we're running the tests through pytest. That > said, I've given up on trying to get pytest-cov to play nicely with this > until I can get a manual invocation of `coverage report ...` to produce > output describing the .pyx files. > > Thanks for reading this far. Got any pointers? > > > ---------------------------------------------------------------- > Context: https://github.com/soltanmm/grpc/tree/cy-wip/src/python > Basic steps to run the tests: > > - cd $GRPC_REPOSITORY_ROOT > - make > - CONFIG=opt ./tools/run_tests/build_python.sh 2.7 > - CONFIG=opt PYVER=2.7 ./run_tests/run_python.sh > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://mail.python.org/pipermail/cython-devel/attachments/20151007/11793594/attachment-0001.html > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > > > ------------------------------ > > End of cython-devel Digest, Vol 57, Issue 5 > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Oct 8 17:35:06 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 8 Oct 2015 17:35:06 +0200 Subject: [Cython] __dealloc__ called even if __cinit__ failed In-Reply-To: <56128374.5010703@cage.ugent.be> References: <56128374.5010703@cage.ugent.be> Message-ID: <56168D2A.3000102@behnel.de> Jeroen Demeyer schrieb am 05.10.2015 um 16:04: > I'm not entirely sure if this is a feature or bug... > > I noticed that, if a __cinit__ call raises an exception on an object, that > __dealloc__ on that same object is still called. > > The problem is that __cinit__ is supposed to set up the C-level attributes > for that object. If this somehow fails, then the object isn't in a > consistent state yet, so it makes no sense to call __dealloc__ on it. > > Do you consider this a Cython bug or should I manually protect my code > against this? In addition to what was already answered, note that __cinit__() is called *after* the CPython instantiation of the class, so the object is fully alive at that point. If the error happens in the middle of an inheritance chain, then the object is created and the upper __cinit__() methods have run, but the lower ones have not and will not be. Since the overall process failed, the object is collected immediately and the __dealloc__() methods are called, all of them, one after the other. Calling no __dealloc__() methods at all would be wrong here and easily leads to resource leaks. But trying to decide in such a case which __dealloc__() methods to call and which not would just make things overly complex and potentially difficult to understand. Calling all of them and letting users guard against uninitialised fields is much simpler, also handles other error cases at the user level and is generally more predictable for all sides. Stefan From jason.madden at nextthought.com Fri Oct 9 18:04:41 2015 From: jason.madden at nextthought.com (Jason Madden) Date: Fri, 9 Oct 2015 11:04:41 -0500 Subject: [Cython] Bug - PyPy tuple leak in __Pyx_PyObject_CallOneArg? Message-ID: <6BDA6804-D691-4B26-B1B7-0395D5E3E9F3@nextthought.com> Hello, I'm one of the maintainers of the gevent concurrency library, which has recently been ported to run on PyPy. Under PyPy, a small portion of the code is compiled with Cython in order to get the desired atomic semantics (specifically, a semaphore class). We recently had a user report an easily reproducible leak of tuples of one element. Tracking it down, it appears that __Pyx_PyObject_CallOneArg creates a new tuple under PyPy, but neglects to free it. This was tested with Cython 0.23.3 and PyPy 2.6.1. Our Cython code contained a loop like this, and every iteration of the loop leaked a tuple: for link in links: link(self) The C output for that last line looked like this: __Pyx_PyObject_CallOneArg(__pyx_t_10, ((PyObject *)__pyx_v_self));... And, under PyPy, the implementation of __Pyx_PyObject_CallOneArg is different than it is under CPython. Specifically, with Cython 0.23.3 this is the code: #if CYTHON_COMPILING_IN_CPYTHON ... #else static CYTHON_INLINE PyObject* __Pyx_PyObject_CallOneArg(PyObject *func, PyObject *arg) { PyObject* args = PyTuple_Pack(1, arg); return (likely(args)) ? __Pyx_PyObject_Call(func, args, NULL) : NULL; } #endif PyTuple_Pack is documented as returning a new reference (https://docs.python.org/2/c-api/tuple.html#c.PyTuple_Pack), so it seems to me like this code should be decrementing the reference to the tuple before returning (much like the __Pyx__PyObject_CallOneArg function does under CPython). Changing the loop to avoid the use of CallOneArg seemed to resolve the tuple leak, presumably at some execution time cost: args = (self,) for link in links: link(*self) There's some additional background at https://bitbucket.org/pypy/pypy/issues/2149/memory-leak-for-python-subclass-of-cpyext#comment-22347393 Am I interpreting this correctly to be a bug, or could there be something wrong in the way we're handling callbacks? Please let me know if there's any further information I can provide. Thanks, Jason From stefan_ml at behnel.de Sat Oct 10 10:07:17 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 10 Oct 2015 10:07:17 +0200 Subject: [Cython] Bug - PyPy tuple leak in __Pyx_PyObject_CallOneArg? In-Reply-To: <6BDA6804-D691-4B26-B1B7-0395D5E3E9F3@nextthought.com> References: <6BDA6804-D691-4B26-B1B7-0395D5E3E9F3@nextthought.com> Message-ID: <5618C735.20004@behnel.de> Jason Madden schrieb am 09.10.2015 um 18:04: > We recently had a user report an easily reproducible leak of tuples of > one element. Tracking it down, it appears that __Pyx_PyObject_CallOneArg > creates a new tuple under PyPy, but neglects to free it. This was tested > with Cython 0.23.3 and PyPy 2.6.1. Thanks for the report and the fix. https://github.com/cython/cython/pull/450 Stefan From stefan_ml at behnel.de Sat Oct 10 14:46:02 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 10 Oct 2015 14:46:02 +0200 Subject: [Cython] Cython 0.23.4 released Message-ID: <5619088A.6050600@behnel.de> Hi everyone, here is a minor bug-fix release for the 0.23 release series that repairs some problems and regressions. https://pypi.python.org/pypi/Cython/0.23.4 Complete changelog follows below. You can get the signed release from here: http://cython.org/ http://cython.org/release/Cython-0.23.4.tar.gz http://cython.org/release/Cython-0.23.4.zip SHA1 sums: fc574c5050cd5a8e34435432e2a4a693353ed807 Cython-0.23.4.tar.gz ad477040a22190b5506a3d8ddde646b7586d5703 Cython-0.23.4.zip Have fun, Stefan 0.23.4 (2015-10-10) =================== Bugs fixed ---------- * Memory leak when calling Python functions in PyPy. * Compilation problem with MSVC in C99-ish mode. * Warning about unused values in a helper macro. From stefan_ml at behnel.de Sat Oct 10 17:27:47 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 10 Oct 2015 17:27:47 +0200 Subject: [Cython] Cython coverage and multiple projects In-Reply-To: References: Message-ID: <56192E73.9030008@behnel.de> Masood Malekghassemi via cython-devel schrieb am 08.10.2015 um 01:06: > I'm having trouble with getting Cython to play nicely with the coverage.py > tool via the Cython.Coverage plug-in. > > We have tests that are running from a different setup.py project than the > one which contains the Cython files (the projects respectively being for > the duration of this discussion the 'tests' project and the 'source' > project). The generated coverage file contains the line data for individual > .pyx files, but the paths of the .pyx files themselves *within* the > .coverage file appear to be rooted in the tests project rather than the > site-packages folder in which the source project's files were installed > (i.e. paths that never exist throughout the edit-build-test cycle). This seems to be a rather unusual setup. The "expected" setting is that tests live somewhere within the source checkout and are run on the non-installed sources, usually as part of the CI build process, sometimes even just as part of a separate reporting process. Why do you want to create coverage reports after the installation? > A secondary issue: the .pyx files are never copied over to the installation > directory, so even if the paths were 'correct' (with respect to being > copied into the site-packages folder or being accessible via developer mode > pip-installs), they wouldn't be found. Sure, that's a difference from .py/.pyc files which are getting executed directly. For Cython code, there is an additional compile/build step that generates the executable code from the sources, which are then no longer needed. That's another reason why support for coverage reporting after the installation seems foreign here and wouldn't be of much use. So, my suggestion would be to run the coverage reports within the source tree after building it, when all source files are readily available. But I'm obviously missing a lot of information about your actual project setup and the reasons behind what you are doing here. Stefan From jtaylor.debian at googlemail.com Sat Oct 10 23:11:37 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sat, 10 Oct 2015 23:11:37 +0200 Subject: [Cython] git master fails to compile scipy Message-ID: <56197F09.90706@googlemail.com> hi, since ab78f93b3ffa88183a0d2aae6b692e394c51f860 scipy does not build anymore: the failing file is: https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_reordering.pyx error: Traceback (most recent call last): File "/usr/bin/cython", line 8, in main(command_line = 1) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", line 704, in main result = compile(sources, options) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", line 679, in compile return compile_multiple(source, options) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", line 657, in compile_multiple result = run_pipeline(source, options, context=context) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", line 487, in run_pipeline err, enddata = Pipeline.run_pipeline(pipeline, source) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py", line 365, in run_pipeline data = phase(data) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py", line 53, in generate_pyx_code_stage module_node.process_implementation(options, result) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py", line 118, in process_implementation self.generate_c_code(env, options, result) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py", line 349, in generate_c_code self.body.generate_function_definitions(env, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 436, in generate_function_definitions stat.generate_function_definitions(env, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 436, in generate_function_definitions stat.generate_function_definitions(env, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/FusedNode.py", line 785, in generate_function_definitions stat.generate_function_definitions(env, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 3068, in generate_function_definitions FuncDefNode.generate_function_definitions(self, env, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 1935, in generate_function_definitions self.generate_function_body(env, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 1691, in generate_function_body self.body.generate_execution_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 442, in generate_execution_code stat.generate_execution_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/UtilNodes.py", line 321, in generate_execution_code self.body.generate_execution_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 6419, in generate_execution_code self.body.generate_execution_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 442, in generate_execution_code stat.generate_execution_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 5865, in generate_execution_code if_clause.generate_execution_code(code, end_label, is_last=i == last) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 5908, in generate_execution_code self.body.generate_execution_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 442, in generate_execution_code stat.generate_execution_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 4800, in generate_execution_code self.generate_assignment_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 5091, in generate_assignment_code self.lhs.generate_assignment_code(self.rhs, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py", line 3897, in generate_assignment_code self.generate_buffer_setitem_code(rhs, code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py", line 3905, in generate_buffer_setitem_code buffer_entry, ptrexpr = self.buffer_lookup_code(code) File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py", line 3873, in buffer_lookup_code for ivar in self.indices] AttributeError: 'NoneType' object has no attribute 'signed' From robertwb at gmail.com Sun Oct 11 07:30:35 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 10 Oct 2015 22:30:35 -0700 Subject: [Cython] git master fails to compile scipy In-Reply-To: <56197F09.90706@googlemail.com> References: <56197F09.90706@googlemail.com> Message-ID: Thanks for the report. This should be fixed with https://github.com/cython/cython/commit/eef2bc650945d69158e888653a2f8bd6bbad3db3 On Sat, Oct 10, 2015 at 2:11 PM, Julian Taylor via cython-devel wrote: > hi, > since ab78f93b3ffa88183a0d2aae6b692e394c51f860 scipy does not build anymore: > the failing file is: > https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_reordering.pyx > > error: > Traceback (most recent call last): > File "/usr/bin/cython", line 8, in > main(command_line = 1) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", > line 704, in main > result = compile(sources, options) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", > line 679, in compile > return compile_multiple(source, options) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", > line 657, in compile_multiple > result = run_pipeline(source, options, context=context) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py", > line 487, in run_pipeline > err, enddata = Pipeline.run_pipeline(pipeline, source) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py", > line 365, in run_pipeline > data = phase(data) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py", > line 53, in generate_pyx_code_stage > module_node.process_implementation(options, result) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py", > line 118, in process_implementation > self.generate_c_code(env, options, result) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py", > line 349, in generate_c_code > self.body.generate_function_definitions(env, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 436, in generate_function_definitions > stat.generate_function_definitions(env, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 436, in generate_function_definitions > stat.generate_function_definitions(env, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/FusedNode.py", > line 785, in generate_function_definitions > stat.generate_function_definitions(env, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 3068, in generate_function_definitions > FuncDefNode.generate_function_definitions(self, env, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 1935, in generate_function_definitions > self.generate_function_body(env, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 1691, in generate_function_body > self.body.generate_execution_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 442, in generate_execution_code > stat.generate_execution_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/UtilNodes.py", > line 321, in generate_execution_code > self.body.generate_execution_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 6419, in generate_execution_code > self.body.generate_execution_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 442, in generate_execution_code > stat.generate_execution_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 5865, in generate_execution_code > if_clause.generate_execution_code(code, end_label, is_last=i == last) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 5908, in generate_execution_code > self.body.generate_execution_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 442, in generate_execution_code > stat.generate_execution_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 4800, in generate_execution_code > self.generate_assignment_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 5091, in generate_assignment_code > self.lhs.generate_assignment_code(self.rhs, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py", > line 3897, in generate_assignment_code > self.generate_buffer_setitem_code(rhs, code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py", > line 3905, in generate_buffer_setitem_code > buffer_entry, ptrexpr = self.buffer_lookup_code(code) > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py", > line 3873, in buffer_lookup_code > for ivar in self.indices] > AttributeError: 'NoneType' object has no attribute 'signed' > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From dalcinl at gmail.com Sun Oct 11 10:30:33 2015 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 11 Oct 2015 11:30:33 +0300 Subject: [Cython] [cython-users] Cython 0.23.4 released In-Reply-To: <5619088A.6050600@behnel.de> References: <5619088A.6050600@behnel.de> Message-ID: On 10 October 2015 at 15:46, Stefan Behnel wrote: > Hi everyone, > > here is a minor bug-fix release for the 0.23 release series that repairs > some problems and regressions. > > https://pypi.python.org/pypi/Cython/0.23.4 > > Complete changelog follows below. > > > You can get the signed release from here: > > http://cython.org/ > > http://cython.org/release/Cython-0.23.4.tar.gz > > http://cython.org/release/Cython-0.23.4.zip > > > SHA1 sums: > fc574c5050cd5a8e34435432e2a4a693353ed807 Cython-0.23.4.tar.gz > ad477040a22190b5506a3d8ddde646b7586d5703 Cython-0.23.4.zip > Stefan, should we upload wheels to PyPI? It is a bit of manual work, as we would need to download artifacts for every build here: https://ci.appveyor.com/project/cython/cython/build/0.23.x-31 then use twine to do the uploads. I do not have the PyPI password to do all by myself. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 From stefan_ml at behnel.de Sun Oct 11 10:42:25 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 Oct 2015 10:42:25 +0200 Subject: [Cython] [cython-users] Cython 0.23.4 released In-Reply-To: References: <5619088A.6050600@behnel.de> Message-ID: <561A20F1.9080604@behnel.de> Lisandro Dalcin schrieb am 11.10.2015 um 10:30: > On 10 October 2015 at 15:46, Stefan Behnel wrote: >> here is a minor bug-fix release for the 0.23 release series that repairs >> some problems and regressions. >> >> https://pypi.python.org/pypi/Cython/0.23.4 > > Stefan, should we upload wheels to PyPI? It is a bit of manual work, > as we would need to download artifacts for every build here: > https://ci.appveyor.com/project/cython/cython/build/0.23.x-31 then use > twine to do the uploads. I do not have the PyPI password to do all by > myself. One of the reasons I released 0.23.4 now (apart from the bug fixes) was to get clean branch builds from which I could upload the wheels. You're almost a day late. :) BTW: $ vanity -q cython Cython has been downloaded 4007918 times! Stefan From jtaylor.debian at googlemail.com Sun Oct 11 12:38:18 2015 From: jtaylor.debian at googlemail.com (Julian Taylor) Date: Sun, 11 Oct 2015 12:38:18 +0200 Subject: [Cython] git master fails to compile scipy In-Reply-To: References: <56197F09.90706@googlemail.com> Message-ID: <561A3C1A.9060002@googlemail.com> thanks, another file fails too: https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_shortest_path.pyx Error compiling Cython file: ------------------------------------------------------------ ... for k from csr_indptr[j] <= k < csr_indptr[j + 1]: w12 = csr_weights[k] ind_k = csr_indices[k] d2 = dist_matrix[i, ind_k] if d1 + w12 < d2: dist_matrix[i, ind_k] = d2 = d1 + w12 ^ ------------------------------------------------------------ scipy/sparse/csgraph/_shortest_path.pyx:798:56: Compiler crash in AnalyseExpressionsTransform ModuleNode.body = StatListNode(_shortest_path.pyx:12:0) StatListNode.stats[17] = StatListNode(_shortest_path.pyx:770:5) StatListNode.stats[0] = CFuncDefNode(_shortest_path.pyx:770:5, args = [...]/6, modifiers = [...]/0, visibility = u'private') File 'Nodes.py', line 430, in analyse_expressions: StatListNode(_shortest_path.pyx:777:4, is_terminator = True) File 'Nodes.py', line 6370, in analyse_expressions: ForFromStatNode(_shortest_path.pyx:786:4, relation1 = u'<=', relation2 = u'<') File 'Nodes.py', line 430, in analyse_expressions: StatListNode(_shortest_path.pyx:787:8) File 'Nodes.py', line 6370, in analyse_expressions: ForFromStatNode(_shortest_path.pyx:790:8, relation1 = u'<=', relation2 = u'<') File 'Nodes.py', line 430, in analyse_expressions: StatListNode(_shortest_path.pyx:791:12) File 'Nodes.py', line 6370, in analyse_expressions: ForFromStatNode(_shortest_path.pyx:791:12, relation1 = u'<=', relation2 = u'<') File 'Nodes.py', line 430, in analyse_expressions: StatListNode(_shortest_path.pyx:792:16) File 'Nodes.py', line 6370, in analyse_expressions: ForFromStatNode(_shortest_path.pyx:793:16, relation1 = u'<=', relation2 = u'<') File 'Nodes.py', line 430, in analyse_expressions: StatListNode(_shortest_path.pyx:794:20) File 'Nodes.py', line 5853, in analyse_expressions: IfStatNode(_shortest_path.pyx:797:20) File 'Nodes.py', line 5899, in analyse_expressions: IfClauseNode(_shortest_path.pyx:797:23) File 'Nodes.py', line 430, in analyse_expressions: StatListNode(_shortest_path.pyx:798:24) File 'Nodes.py', line 4787, in analyse_expressions: CascadedAssignmentNode(_shortest_path.pyx:798:56, assignment_overloads = [...]/0) File 'Nodes.py', line 5157, in analyse_types: CascadedAssignmentNode(_shortest_path.pyx:798:56, assignment_overloads = [...]/0) Compiler crash traceback from this point on: File "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", line 5157, in analyse_types overloaded = lhs.type.is_cpp_class and env.lookup_operator('=', [lhs, self.rhs]) AttributeError: 'NoneType' object has no attribute 'is_cpp_class' On 11.10.2015 07:30, Robert Bradshaw wrote: > Thanks for the report. This should be fixed with > https://github.com/cython/cython/commit/eef2bc650945d69158e888653a2f8bd6bbad3db3 > > On Sat, Oct 10, 2015 at 2:11 PM, Julian Taylor via cython-devel > wrote: >> hi, >> since ab78f93b3ffa88183a0d2aae6b692e394c51f860 scipy does not build anymore: >> the failing file is: >> https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_reordering.pyx >> From stefan_ml at behnel.de Sun Oct 11 16:30:22 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 Oct 2015 16:30:22 +0200 Subject: [Cython] can we deprecate for-from loops? Message-ID: <561A727E.1050402@behnel.de> Hi! The syntax construct "for i from 0 <= i < 10" has been silently outdated for years. Can we start issuing a warning that normal range() loops are preferred? While I don't see it going to go away any time soon, I'd like to make users aware that it's an unnecessary syntax wart that's best avoided and that should really not be used in new code. Stefan From stefan_ml at behnel.de Sun Oct 11 17:32:45 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 11 Oct 2015 17:32:45 +0200 Subject: [Cython] git master fails to compile scipy In-Reply-To: <561A3C1A.9060002@googlemail.com> References: <56197F09.90706@googlemail.com> <561A3C1A.9060002@googlemail.com> Message-ID: <561A811D.7040309@behnel.de> Julian Taylor via cython-devel schrieb am 11.10.2015 um 12:38: > another file fails too: > https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_shortest_path.pyx > > Error compiling Cython file: > ------------------------------------------------------------ > ... > for k from csr_indptr[j] <= k < csr_indptr[j + 1]: > w12 = csr_weights[k] > ind_k = csr_indices[k] > d2 = dist_matrix[i, ind_k] > if d1 + w12 < d2: > dist_matrix[i, ind_k] = d2 = d1 + w12 > ^ > ------------------------------------------------------------ > > scipy/sparse/csgraph/_shortest_path.pyx:798:56: Compiler crash in > AnalyseExpressionsTransform > > ModuleNode.body = StatListNode(_shortest_path.pyx:12:0) > StatListNode.stats[17] = StatListNode(_shortest_path.pyx:770:5) > StatListNode.stats[0] = CFuncDefNode(_shortest_path.pyx:770:5, > args = [...]/6, > modifiers = [...]/0, > visibility = u'private') > File 'Nodes.py', line 430, in analyse_expressions: > StatListNode(_shortest_path.pyx:777:4, > is_terminator = True) > File 'Nodes.py', line 6370, in analyse_expressions: > ForFromStatNode(_shortest_path.pyx:786:4, > relation1 = u'<=', > relation2 = u'<') > File 'Nodes.py', line 430, in analyse_expressions: > StatListNode(_shortest_path.pyx:787:8) > File 'Nodes.py', line 6370, in analyse_expressions: > ForFromStatNode(_shortest_path.pyx:790:8, > relation1 = u'<=', > relation2 = u'<') > File 'Nodes.py', line 430, in analyse_expressions: > StatListNode(_shortest_path.pyx:791:12) > File 'Nodes.py', line 6370, in analyse_expressions: > ForFromStatNode(_shortest_path.pyx:791:12, > relation1 = u'<=', > relation2 = u'<') > File 'Nodes.py', line 430, in analyse_expressions: > StatListNode(_shortest_path.pyx:792:16) > File 'Nodes.py', line 6370, in analyse_expressions: > ForFromStatNode(_shortest_path.pyx:793:16, > relation1 = u'<=', > relation2 = u'<') > File 'Nodes.py', line 430, in analyse_expressions: > StatListNode(_shortest_path.pyx:794:20) > File 'Nodes.py', line 5853, in analyse_expressions: > IfStatNode(_shortest_path.pyx:797:20) > File 'Nodes.py', line 5899, in analyse_expressions: > IfClauseNode(_shortest_path.pyx:797:23) > File 'Nodes.py', line 430, in analyse_expressions: > StatListNode(_shortest_path.pyx:798:24) > File 'Nodes.py', line 4787, in analyse_expressions: > CascadedAssignmentNode(_shortest_path.pyx:798:56, > assignment_overloads = [...]/0) > File 'Nodes.py', line 5157, in analyse_types: > CascadedAssignmentNode(_shortest_path.pyx:798:56, > assignment_overloads = [...]/0) > > Compiler crash traceback from this point on: > File > "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py", > line 5157, in analyse_types > overloaded = lhs.type.is_cpp_class and env.lookup_operator('=', > [lhs, self.rhs]) > AttributeError: 'NoneType' object has no attribute 'is_cpp_class' Thanks, here's a fix: https://github.com/cython/cython/commit/ec1c92ee169e014cc12e5f3ef99d5ec0150d3228 There was a major refactoring in the latest master that cleaned up all the convoluted indexing and buffer/memoryview access code that piled up over the years. I'm not too surprised that I missed a few places and broke some code along the way. Thanks for testing. Stefan From jdemeyer at cage.ugent.be Sun Oct 11 22:01:58 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Sun, 11 Oct 2015 22:01:58 +0200 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <561A727E.1050402@behnel.de> References: <561A727E.1050402@behnel.de> Message-ID: <561AC036.9080300@cage.ugent.be> On 2015-10-11 16:30, Stefan Behnel wrote: > Hi! > > The syntax construct "for i from 0 <= i < 10" has been silently outdated > for years. Can we start issuing a warning that normal range() loops are > preferred? Is there anything wrong with the for ... from ... syntax? In more complicated situations I find for i from a <= i < b by s more readable than the equivalent for i in range() statement. I know it's not a strong argument, it's just something to keep in mind. From wstein at gmail.com Sun Oct 11 23:46:49 2015 From: wstein at gmail.com (William Stein) Date: Sun, 11 Oct 2015 14:46:49 -0700 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <561AC036.9080300@cage.ugent.be> References: <561A727E.1050402@behnel.de> <561AC036.9080300@cage.ugent.be> Message-ID: On Sun, Oct 11, 2015 at 1:01 PM, Jeroen Demeyer wrote: > On 2015-10-11 16:30, Stefan Behnel wrote: >> >> Hi! >> >> The syntax construct "for i from 0 <= i < 10" has been silently outdated >> for years. Can we start issuing a warning that normal range() loops are >> preferred? > > > Is there anything wrong with the for ... from ... syntax? In more > complicated situations I find > > for i from a <= i < b by s > > more readable than the equivalent for i in range() statement. > > I know it's not a strong argument, it's just something to keep in mind. A weak argument for removing the " for i from a <= i < b by s" syntax: - it makes it more difficult for somebody who only knows Python to read Cython code (adding one new idea that is not necessary). - it doesn't look Pythonic (purely subjective) -- William > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel -- William (http://wstein.org) From greg.ewing at canterbury.ac.nz Mon Oct 12 00:31:42 2015 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Oct 2015 11:31:42 +1300 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <561A727E.1050402@behnel.de> References: <561A727E.1050402@behnel.de> Message-ID: <561AE34E.1090700@canterbury.ac.nz> Stefan Behnel wrote: > Hi! > > The syntax construct "for i from 0 <= i < 10" has been silently outdated > for years. Can we start issuing a warning that normal range() loops are > preferred? I'd be in favour of replacing it with just 'for 0 <= i < 10', but -1 on removing it altogether. I introduced it in Pyrex for a reason -- to clearly express iterations over ranges of integers with arbitrary combinations of open/closed endpoints, for use in conjunction with C code. I believe that reason is still valid. -- Greg From nykevin.norris at gmail.com Mon Oct 12 04:57:29 2015 From: nykevin.norris at gmail.com (Kevin Norris) Date: Sun, 11 Oct 2015 22:57:29 -0400 Subject: [Cython] can we deprecate for-from loops? Message-ID: Greg Ewing wrote: > Stefan Behnel wrote: > > Hi! > > > > The syntax construct "for i from 0 <= i < 10" has been silently outdated > > for years. Can we start issuing a warning that normal range() loops are > > preferred? > > I'd be in favour of replacing it with just 'for 0 <= i < 10', > but -1 on removing it altogether. > > I introduced it in Pyrex for a reason -- to clearly express > iterations over ranges of integers with arbitrary combinations > of open/closed endpoints, for use in conjunction with C code. > I believe that reason is still valid. > > -- > Greg IMHO, either it should be un-deprecated in the documentation[1], or the compiler should start issuing warnings and eventually errors. You can't have it both ways. Either it's deprecated or it isn't. [1]: http://docs.cython.org/src/reference/language_basics.html#for-loops -- Kevin From jdemeyer at cage.ugent.be Mon Oct 12 16:52:59 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Mon, 12 Oct 2015 16:52:59 +0200 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: References: Message-ID: <561BC94B.5000805@cage.ugent.be> On 2015-10-12 04:57, Kevin Norris wrote: > IMHO, either it should be un-deprecated in the documentation[1], or > the compiler should start issuing warnings and eventually errors. You > can't have it both ways. Either it's deprecated or it isn't. +1 From robertwb at gmail.com Mon Oct 12 17:37:18 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 12 Oct 2015 08:37:18 -0700 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <561AE34E.1090700@canterbury.ac.nz> References: <561A727E.1050402@behnel.de> <561AE34E.1090700@canterbury.ac.nz> Message-ID: On Sun, Oct 11, 2015 at 3:31 PM, Greg Ewing wrote: > Stefan Behnel wrote: >> >> Hi! >> >> The syntax construct "for i from 0 <= i < 10" has been silently outdated >> for years. Can we start issuing a warning that normal range() loops are >> preferred? > > > I'd be in favour of replacing it with just 'for 0 <= i < 10', > but -1 on removing it altogether. > > I introduced it in Pyrex for a reason -- to clearly express > iterations over ranges of integers with arbitrary combinations > of open/closed endpoints, I agree that it expresses intent clearer than the range(...) construct. > for use in conjunction with C code. > I believe that reason is still valid. I'm not sure why "in conjunction with C code" makes any difference to the argument, now that the basic range loop produces the same code in Cython. Put another way, does its clarity merit a PEP to introduce this syntax into Python? If not, it's hard to justify in Cython given that it is a fully redundant addition to the language. - Robert From josh at ayers.pw Mon Oct 12 20:40:54 2015 From: josh at ayers.pw (Josh Ayers) Date: Mon, 12 Oct 2015 11:40:54 -0700 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: References: <561A727E.1050402@behnel.de> <561AE34E.1090700@canterbury.ac.nz> Message-ID: <1444675254.617443.408179969.28E7422A@webmail.messagingengine.com> When the step size is a variable, the range statement falls back to a Python for loop since the direction of the loop is unknown. The "i from a <= i < b by c" syntax specifies the direction, so it is turned into a C for loop. The difference is performance could be substantial. This produces a Python for loop: def python_loop(Py_ssize_t a, Py_ssize_t b, Py_ssize_t c): cdef Py_ssize_t i for i in range(a, b, c): pass This produces a C for loop: def c_loop(Py_ssize_t a, Py_ssize_t b, Py_ssize_t c): cdef Py_ssize_t i for i from a <= i < b by c: pass On Mon, Oct 12, 2015, at 08:37 AM, Robert Bradshaw wrote: > On Sun, Oct 11, 2015 at 3:31 PM, Greg Ewing > wrote: > > Stefan Behnel wrote: > >> > >> Hi! > >> > >> The syntax construct "for i from 0 <= i < 10" has been silently outdated > >> for years. Can we start issuing a warning that normal range() loops are > >> preferred? > > > > > > I'd be in favour of replacing it with just 'for 0 <= i < 10', > > but -1 on removing it altogether. > > > > I introduced it in Pyrex for a reason -- to clearly express > > iterations over ranges of integers with arbitrary combinations > > of open/closed endpoints, > > I agree that it expresses intent clearer than the range(...) construct. > > > for use in conjunction with C code. > > I believe that reason is still valid. > > I'm not sure why "in conjunction with C code" makes any difference to > the argument, now that the basic range loop produces the same code in > Cython. Put another way, does its clarity merit a PEP to introduce > this syntax into Python? If not, it's hard to justify in Cython given > that it is a fully redundant addition to the language. > > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From robertwb at gmail.com Tue Oct 13 06:51:22 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 12 Oct 2015 21:51:22 -0700 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <1444675254.617443.408179969.28E7422A@webmail.messagingengine.com> References: <561A727E.1050402@behnel.de> <561AE34E.1090700@canterbury.ac.nz> <1444675254.617443.408179969.28E7422A@webmail.messagingengine.com> Message-ID: On Mon, Oct 12, 2015 at 11:40 AM, Josh Ayers wrote: > When the step size is a variable, the range statement falls back to a > Python for loop since the direction of the loop is unknown. The "i from > a <= i < b by c" syntax specifies the direction, so it is turned into a > C for loop. The difference is performance could be substantial. > > This produces a Python for loop: > > def python_loop(Py_ssize_t a, Py_ssize_t b, Py_ssize_t c): > cdef Py_ssize_t i > for i in range(a, b, c): > pass I thought we had fixed that... apparently not. Doable (e.g. multiply a, b, c, and then i by the sign of c, checking for 0 of course) but messy. > This produces a C for loop: > > def c_loop(Py_ssize_t a, Py_ssize_t b, Py_ssize_t c): > cdef Py_ssize_t i > for i from a <= i < b by c: > pass Ah, there is a slight semantic difference: this will be an infinite for...from loop if c is not positive (and a < b), but an empty for...in loop for range. However, there's another catch: for...from can be used with non-integral values (doubles, pointers, ...) Unless we wanted to introduce, say, something like cython.crange it may be more painful to force users to express that as while loops. I was hoping to get rid of it, but now I'm not sure it'd be that easy. > On Mon, Oct 12, 2015, at 08:37 AM, Robert Bradshaw wrote: >> On Sun, Oct 11, 2015 at 3:31 PM, Greg Ewing >> wrote: >> > Stefan Behnel wrote: >> >> >> >> Hi! >> >> >> >> The syntax construct "for i from 0 <= i < 10" has been silently outdated >> >> for years. Can we start issuing a warning that normal range() loops are >> >> preferred? >> > >> > >> > I'd be in favour of replacing it with just 'for 0 <= i < 10', >> > but -1 on removing it altogether. >> > >> > I introduced it in Pyrex for a reason -- to clearly express >> > iterations over ranges of integers with arbitrary combinations >> > of open/closed endpoints, >> >> I agree that it expresses intent clearer than the range(...) construct. >> >> > for use in conjunction with C code. >> > I believe that reason is still valid. >> >> I'm not sure why "in conjunction with C code" makes any difference to >> the argument, now that the basic range loop produces the same code in >> Cython. Put another way, does its clarity merit a PEP to introduce >> this syntax into Python? If not, it's hard to justify in Cython given >> that it is a fully redundant addition to the language. >> >> - Robert >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From jwilk at jwilk.net Thu Oct 15 23:26:19 2015 From: jwilk at jwilk.net (Jakub Wilk) Date: Thu, 15 Oct 2015 23:26:19 +0200 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <561A727E.1050402@behnel.de> References: <561A727E.1050402@behnel.de> Message-ID: <20151015212619.GA1873@jwilk.net> * Stefan Behnel , 2015-10-11, 16:30: >The syntax construct "for i from 0 <= i < 10" has been silently >outdated for years. Can we start issuing a warning that normal range() >loops are preferred? Hmm. AFAICS, Cython doesn't aways optimize range() loops that well... For the attached example code, Cython 0.23.4 generated nice for loops for the for_from() function: for (__pyx_t_1 = 0; __pyx_t_1 < 10; __pyx_t_1++) { ... for (__pyx_t_3 = 0; __pyx_t_3 < 10; __pyx_t_3++) { ... but not for the for_range() function: __pyx_t_1 = __Pyx_PyObject_Call(__pyx_builtin_range, __pyx_tuple_, NULL); ... ... (The problem goes away if cdef the iteration variables, which I should do anyway, so it's not a big deal for me.) -- Jakub Wilk -------------- next part -------------- def for_from(): for i from 0 <= i < 10: for j from 0 <= j < 10: print(i + j) def for_range(): for i in range(10): for j in range(10): print(i + j) From robertwb at math.washington.edu Fri Oct 16 01:51:42 2015 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 15 Oct 2015 16:51:42 -0700 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <20151015212619.GA1873@jwilk.net> References: <561A727E.1050402@behnel.de> <20151015212619.GA1873@jwilk.net> Message-ID: On Thu, Oct 15, 2015 at 2:26 PM, Jakub Wilk wrote: > * Stefan Behnel , 2015-10-11, 16:30: >> >> The syntax construct "for i from 0 <= i < 10" has been silently outdated >> for years. Can we start issuing a warning that normal range() loops are >> preferred? > > > Hmm. AFAICS, Cython doesn't aways optimize range() loops that well... For > the attached example code, Cython 0.23.4 generated nice for loops for the > for_from() function: > > for (__pyx_t_1 = 0; __pyx_t_1 < 10; __pyx_t_1++) { > ... > for (__pyx_t_3 = 0; __pyx_t_3 < 10; __pyx_t_3++) { > ... > > but not for the for_range() function: > > __pyx_t_1 = __Pyx_PyObject_Call(__pyx_builtin_range, __pyx_tuple_, NULL); > ... > ... > > (The problem goes away if cdef the iteration variables, which I should do > anyway, so it's not a big deal for me.) Correct, if you don't cdef the iteration (or bounds) variable it doesn't know what size of int to use (e.g. you could be doing range(10**100, 10**100 + 10). Also, as you point out, the overhead of using a Python target variable (for either format) is larger than range vs. C for loop. From atash at google.com Fri Oct 16 02:58:33 2015 From: atash at google.com (Masood Malekghassemi) Date: Thu, 15 Oct 2015 17:58:33 -0700 Subject: [Cython] Cython coverage and multiple projects In-Reply-To: <56192E73.9030008@behnel.de> References: <56192E73.9030008@behnel.de> Message-ID: Given that we have control over how and where source files get used by the virtualenv, your suggestion seems within reach by doing a bit of temporary-directory dancing. I've been hitting some issues with that (unrelated to Cython as far as I can tell), and will attempt to update this thread after digging. Some more background: we'd been trying to avoid running tests directly within the source directory and in a separate package to 'cleanly' capture how a user of the library would see the code (from grabbing it e.g. on PyPI to using it) and to keep the user free of the test modules on installation, respectively. The ideals are gradually getting, uh, *influenced* by trying to keep a variety of tools happy. It may be that the ideals simply aren't meant to be (though certainly the separate-package reasoning may be solvable via a host of other means, e.g. mucking around with the manifest if the install->test reasoning is scrapped). At the risk of bleeding frustration onto the page: a myriad of bugs, 'features', and such across multiple tools have turned this into an exhausting grid search across axes among our desiderata (e.g. parallel test execution), test tools, test setups, coverage setups, project organization, test plugins, and varying degrees of separation of responsibility between some of our shim scripts and the tools themselves. Certainly this is not a problem originating from Cython, but I do wish there'd been some complete blessed example project demonstrating the basic coverage+testing setup along the way for Cython, possibly with some glue to e.g. py.test or nosetests or nose2 or even a custom distutils/setuptools test command etc.. Thanks for the response! On Sat, Oct 10, 2015 at 8:27 AM, Stefan Behnel wrote: > Masood Malekghassemi via cython-devel schrieb am 08.10.2015 um 01:06: > > I'm having trouble with getting Cython to play nicely with the > coverage.py > > tool via the Cython.Coverage plug-in. > > > > We have tests that are running from a different setup.py project than the > > one which contains the Cython files (the projects respectively being for > > the duration of this discussion the 'tests' project and the 'source' > > project). The generated coverage file contains the line data for > individual > > .pyx files, but the paths of the .pyx files themselves *within* the > > .coverage file appear to be rooted in the tests project rather than the > > site-packages folder in which the source project's files were installed > > (i.e. paths that never exist throughout the edit-build-test cycle). > > This seems to be a rather unusual setup. The "expected" setting is that > tests live somewhere within the source checkout and are run on the > non-installed sources, usually as part of the CI build process, sometimes > even just as part of a separate reporting process. Why do you want to > create coverage reports after the installation? > > > > A secondary issue: the .pyx files are never copied over to the > installation > > directory, so even if the paths were 'correct' (with respect to being > > copied into the site-packages folder or being accessible via developer > mode > > pip-installs), they wouldn't be found. > > Sure, that's a difference from .py/.pyc files which are getting executed > directly. For Cython code, there is an additional compile/build step that > generates the executable code from the sources, which are then no longer > needed. That's another reason why support for coverage reporting after the > installation seems foreign here and wouldn't be of much use. > > So, my suggestion would be to run the coverage reports within the source > tree after building it, when all source files are readily available. But > I'm obviously missing a lot of information about your actual project setup > and the reasons behind what you are doing here. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Oct 16 08:26:26 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 16 Oct 2015 08:26:26 +0200 Subject: [Cython] can we deprecate for-from loops? In-Reply-To: <20151015212619.GA1873@jwilk.net> References: <561A727E.1050402@behnel.de> <20151015212619.GA1873@jwilk.net> Message-ID: <56209892.2000006@behnel.de> Jakub Wilk schrieb am 15.10.2015 um 23:26: > * Stefan Behnel, 2015-10-11, 16:30: >> The syntax construct "for i from 0 <= i < 10" has been silently outdated >> for years. Can we start issuing a warning that normal range() loops are >> preferred? > > Hmm. AFAICS, Cython doesn't aways optimize range() loops that well... For > the attached example code, Cython 0.23.4 generated nice for loops for the > for_from() function: > > for (__pyx_t_1 = 0; __pyx_t_1 < 10; __pyx_t_1++) { > ... > for (__pyx_t_3 = 0; __pyx_t_3 < 10; __pyx_t_3++) { > ... > > but not for the for_range() function: > > __pyx_t_1 = __Pyx_PyObject_Call(__pyx_builtin_range, __pyx_tuple_, NULL); ... > ... Your example was: """ def for_from(): for i from 0 <= i < 10: for j from 0 <= j < 10: print(i + j) def for_range(): for i in range(10): for j in range(10): print(i + j) """ The reason why i and j are not automatically being inferred as C long variables (although the range obviously fits into a long) is not the range loop but the fact that afterwards you are doing "i + j". Cython doesn't see their original value range at that point any more and thus cannot determine that this operation will not overflow in C long space, so it takes the safe route and makes both Python objects. If instead you did for i in range(10): for j in range(10): print(i) print(j) then Cython would consider C long a safe choice for i and j. You can get the same effect by setting the directive "infer_types=True", which enables full (unsafe) type inference. BTW, my guess is that in the first example, only the C for-loop is executed in C space. The values would be converted to a Python object right afterwards, which, as Robert said, pretty much ruins the advantage. Stefan From carlosjosepita at gmail.com Fri Oct 16 20:53:01 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Fri, 16 Oct 2015 15:53:01 -0300 Subject: [Cython] [Bug] Generator comprehension not compiling Message-ID: Hi, in cython 0.24.0a0 this: ---------- import cython as cy @cy.cfunc def f(cats): ''.join(c for c in cats) ---------- fails to compile with: Compiler crash in AnalyseDeclarationsTransform. Nevertheless the following close variations do compile: ---------- import cython as cy @cy.cfunc def f(cats): ''.join([c for c in cats]) ---------- ---------- cdef f(cats): ''.join(c for c in cats) ---------- Cheers -- Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosjosepita at gmail.com Fri Oct 16 21:05:10 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Fri, 16 Oct 2015 16:05:10 -0300 Subject: [Cython] [Bug] Unable to cast to python type. Message-ID: Hi, import cython as cy y = cy.cast('list', x) fails to compile with "AttributeError: 'TypecastNode' object has no attribute 'typecheck'". But the following examples do compile: y = x y = cy.cast('int', x) Cheers -- Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From leandro at limaesilva.com.br Fri Oct 16 16:04:33 2015 From: leandro at limaesilva.com.br (Leandro Pereira de Lima e Silva) Date: Fri, 16 Oct 2015 11:04:33 -0300 Subject: [Cython] Bug in list comprehension Message-ID: Hello, I've think I've found a bug in Cython (or is this just a funny feature?). The error occurs when the variable name used as source for the iteration is also used as a value. Given the files test1.pyx (cython) and test2.py (python): test1.pyx: def test(): x = range(10) return [x for x in x] test2.py: def test(): x = range(10) return [x for x in x] The results in running them are: >>> import test1, test2 >>> test1.test() Traceback (most recent call last): File "", line 1, in File "test1.pyx", line 4, in test1.test (/Users/leandro/.pyxbld/temp.macosx-10.6-intel-3.4/pyrex/test1.c:666) return [x for x in x] UnboundLocalError: local variable 'x' referenced before assignment >>> test2.test() [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> A regular for loop will behave correctly: code: def test(): x = range(10) ret = [] for x in x: ret.append(x) return ret result: >>> import test1, test2 >>> test1.test() [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> test2.test() [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> Cheers, Leandro -------------- next part -------------- An HTML attachment was scrubbed... URL: From krthornt at uci.edu Fri Oct 16 21:50:41 2015 From: krthornt at uci.edu (Kevin Thornton) Date: Fri, 16 Oct 2015 19:50:41 +0000 Subject: [Cython] Exposing C++'s Message-ID: Hello, I was wanting to use C++s' back_inserter in Cython, and I got it working. I'm writing the list to ask about what more I should do before submitting a pull request. In short, the following definitions were sufficient: ##Define the basic iterator types (this is incomplete) cdef extern from "" namespace "std": cdef cppclass iterator[Category,T,Distance,Pointer,Reference]: pass cdef cppclass output_iterator_tag: pass #The goal here is to get back_inserter working, as it is handy. cdef cppclass back_insert_iterator[T](iterator[output_iterator_tag,void,void,void,void]): pass back_insert_iterator[CONTAINER] back_inserter[CONTAINER](CONTAINER &) ##Not shown here, but I've gotten C++11's move algorithm to work, too, ##using same kind of prototype cdef extern from "" namespace "std": OUTPUT copy[INPUT,OUTPUT](INPUT,INPUT,OUTPUT) I've worked up a tiny test package that is on GitHub: https://github.com/molpopgen/cython_cpp_iterators The tests work, and I can indeed copy amongst C++ containers. I've also gotten the C++ move algorithm to work. The prototype is the same as for 'copy' above. I'd be willing to define the remaining classes from and add a .pxd file. Any input would be appreciated. Thanks, Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Oct 17 08:18:25 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 17 Oct 2015 08:18:25 +0200 Subject: [Cython] Bug in list comprehension In-Reply-To: References: Message-ID: <5621E831.4090105@behnel.de> Leandro Pereira de Lima e Silva schrieb am 16.10.2015 um 16:04: > I've think I've found a bug in Cython (or is this just a funny feature?). > The error occurs when the variable name used as source for the iteration is > also used as a value. > > Given the files test1.pyx (cython) and test2.py (python): > > test1.pyx: > def test(): > x = range(10) > return [x for x in x] > > test2.py: > def test(): > x = range(10) > return [x for x in x] Note that you could just compile test2.py in Cython. No need to change the suffix. > The results in running them are: > >>> import test1, test2 > >>> test1.test() > Traceback (most recent call last): > File "", line 1, in > File "test1.pyx", line 4, in test1.test > (/Users/leandro/.pyxbld/temp.macosx-10.6-intel-3.4/pyrex/test1.c:666) > return [x for x in x] > UnboundLocalError: local variable 'x' referenced before assignment Yes, that's a known bug. Pops up every once in a while, difficult to fix, easy to work around. http://trac.cython.org/ticket/600 Stefan From carlosjosepita at gmail.com Sun Oct 18 16:39:41 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 18 Oct 2015 11:39:41 -0300 Subject: [Cython] [Bug] Unable to cast to python type. In-Reply-To: References: Message-ID: I've been trying to understand the problem here. The cause is easy to explain, for the cure I would like to consult the savvy before PR'ing. The cause: * Parsing of '<...>' will instantiate a typecast from a basetype node and a declarator node, then the typecast type will be inferred from these two nodes. * ParseTreeTransform of 'cast(....)' will instead instantiate a typecast node directly from a type (parsed as 'sizeof(...)'). So there will be no self.base_type in this part of TypecastNode.analyse_types: if to_py and not from_py: .... elif self.operand.type.can_coerce_to_pyobject(env): self.result_ctype = py_object_type base_type = self.base_type.analyse(env) self.operand = self.operand.coerce_to(base_type, env) .... The question: Why isn't the last line simply "self.operand = self.operand.coerce_to(self.type, env)"? That is, what's the point of coercing to the base type here instead of coercing to the type itself? A fortiori, notice this is inside the to_py case, I'm not even sure that base_type makes sense here. For example: - Pointer base type cannot be a Python object (for ). - Const base type cannot be a Python object (for ). - Cannot cast to a function type (for ). (the compiler dixit) Cheers -- Carlos On Fri, Oct 16, 2015 at 4:05 PM, Carlos Pita wrote: > Hi, > > import cython as cy > y = cy.cast('list', x) > > fails to compile with "AttributeError: 'TypecastNode' object has no > attribute 'typecheck'". > > But the following examples do compile: > > y = x > > y = cy.cast('int', x) > > Cheers > -- > Carlos > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosjosepita at gmail.com Sun Oct 18 19:45:56 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 18 Oct 2015 14:45:56 -0300 Subject: [Cython] [Refactor] Emulated types Message-ID: Hi, I'm in the process of refactoring the emulated types in Shadow, in order to simplify a bit that part of the module. Ultimately I want to add support for declaring typed memory views in pure python (currently it's only possible to declare them as strings, Shadow does implement the slicing part -tho it's messy here- but the parser is unable to get them right from decorators). But before that I'd like to simplify things a bit, as I've said, and before that I need to get a good grasp of what I'm trying to simplify, so I'd better hear your opinion about the following points first: 1) Shadow allows creating array types like this: cython.array(cython.int, 10) but the compiler will complain that cython.array is an unknown type. 2) Furthermore Shadow allows to instantiate every type: cython.int(10), cython.array(cython.int, 20)(), etc. The compiler rejects all these constructs, so I see little point in supporting this feature. I think that disallowing these features (AFAICS completely unsupported by the compiler) will simplify the code a lot and will allow to merge the "shameless copy" section in a more coherent fashion. Is there any compelling reason to keep them? What is/was their rationale? Cheers -- Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosjosepita at gmail.com Sun Oct 18 22:15:17 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 18 Oct 2015 17:15:17 -0300 Subject: [Cython] [Refactor] Emulated types In-Reply-To: References: Message-ID: Maybe I'm misunderstanding the entire point, but this perplexes me: 3) cython.address will create a pointer from a list, another pointer or an instance of ArrayType (although, as I commented before: I see no compiler-compatible way to instantiate an ArrayType). In any case the list or array will be a (type casted) *copy* of the original one, thus the reference semantics of pointers would be lost here. This could be workarounded if the cast was replaced by a check and then the original (type checked) list or array was kept *referenced* from inside the pointer object. But, nevertheless, I see no general way to preserve reference semantics in pure python mode, since there are immutable objects in python. What is to be done with ints, for example? (notice this is the example for cython.address in the docs), so maybe cython.address should be banned for good, or at least restricted. If it's just restricted I think that instead of trying to emulate some cases (say lists/arrays) and disallow others (say ints/floats) it could be saner to just keep pointers opaque. In terms of the current implementation, that would imply to remove __setitem__, __getitem__ and initialization from objects other than pointers or None/0. I don't know... maybe this __setitem__, __getitem__, __init__(from list/array) stuff is there just for the sake of the ArrayType subclass. But then it would be better to push it down the hierarchy and, again, I still don't get how to instantiate ArrayType in a way that the compiler likes, so I feel more inclined to remove it. Sorry if I'm missing the point. Cheers -- Carlos On Sun, Oct 18, 2015 at 2:45 PM, Carlos Pita wrote: > Hi, > > I'm in the process of refactoring the emulated types in Shadow, in order > to simplify a bit that part of the module. Ultimately I want to add support > for declaring typed memory views in pure python (currently it's only > possible to declare them as strings, Shadow does implement the slicing part > -tho it's messy here- but the parser is unable to get them right from > decorators). But before that I'd like to simplify things a bit, as I've > said, and before that I need to get a good grasp of what I'm trying to > simplify, so I'd better hear your opinion about the following points first: > > 1) Shadow allows creating array types like this: cython.array(cython.int, > 10) but the compiler will complain that cython.array is an unknown type. > > 2) Furthermore Shadow allows to instantiate every type: cython.int(10), > cython.array(cython.int, 20)(), etc. The compiler rejects all these > constructs, so I see little point in supporting this feature. > > I think that disallowing these features (AFAICS completely unsupported by > the compiler) will simplify the code a lot and will allow to merge the > "shameless copy" section in a more coherent fashion. Is there any > compelling reason to keep them? What is/was their rationale? > > Cheers > -- > Carlos > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosjosepita at gmail.com Sun Oct 18 22:35:44 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 18 Oct 2015 17:35:44 -0300 Subject: [Cython] [Refactor] Emulated types In-Reply-To: References: Message-ID: > keep pointers opaque... that would imply to remove __setitem__, __getitem__,... Probably I went too far there: at least ptr[0] could be kept as a dereference operator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosjosepita at gmail.com Mon Oct 19 00:31:06 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 18 Oct 2015 19:31:06 -0300 Subject: [Cython] [Refactor] Emulated types In-Reply-To: References: Message-ID: > > 3) cython.address will create a pointer from a list, another pointer or an > instance of ArrayType (although, as I commented before: I see no > compiler-compatible way to instantiate an ArrayType). > I just realized this is not an accurate description of what cython.address does, but of what PointerType does instead. I hadn't noticed the [] boxing cython.address does when creating the pointer. So I guess the list case in the PointerType constructor is there just to support this boxed parameter, not lists per se. Nevertheless my main concerns still stand: (i) the boxing won't make cython.address(x) equivalent to &x when x is, say, an int (you get a "pointer to the box", not a "pointer to the int"); (ii) I don't know how to create an instance of ArrayType in a way the compiler likes (and I don't understand the point of doing that, even if it were possible) and (iii) in any case that array will be copied when the PointerType constructor type casts it, so I won't really get a pointer to an array but a new array instead. Sorry for the autoanswering, I'm a bit overwhelmed with my first intimate contact with cython. Any clarifications will be *very much* appreciated. Regards -- Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Wed Oct 21 17:52:30 2015 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Wed, 21 Oct 2015 08:52:30 -0700 Subject: [Cython] 'Class' redeclared error if C++ class & file name match those of the wrapper class Message-ID: Hi all, I'm forwarding this at the request of David Nemesky ("savior" on Freenode), who I've been chatting with on the cython IRC channel. He was having trouble signing up to the cython-devel mailing list, so I've gone ahead and sent his bug report in for him. His report follows below. Apparently this error happens under very specific circumstances, but I managed to run into them. When wrapping a C++ class, Cython reports a "'ClassName' redeclared" error, if: - the name of the .pyx is the same as the name of the .pxd - the name of the python class is the same as the name of the C++ class Mismatch in either the file- or the class name results in a successful compilation. I have modified my fork of the cython-cpp-test repo to make the issue easily reproducible. In https://github.com/DavidNemeskey/cython-cpp-test/, try `python setup.py build_ext --inplace` (do not forget to delete all .cpp and .so files between calls) in the following branches: - cython_bug (diff file, diff class) -> OK - diff_file_same_class -> OK - same_file_diff_class -> OK - same_file_same_class -> ERROR (https://bpaste.net/show/2c1af5fa5ee2) Since Cython compiles xxx.pyx to xxx.cpp in this case, in order to avoid problems with testclass.cpp being overwritten, I implemented the class in testclass.hpp (in the original repo, there was a .h and a .cpp file). However, implementing everything in the header is not a pre-requisite of the bug, as is evident from branch same_file_same_class_diff_c. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thouis at gmail.com Thu Oct 22 04:22:15 2015 From: thouis at gmail.com (Thouis (Ray) Jones) Date: Wed, 21 Oct 2015 22:22:15 -0400 Subject: [Cython] keyword argument to prange() for num_threads gives undefined error Message-ID: I couldn't find this in the Cython Trac, but forgive me if it's been addressed already. One of my students ran into the bug referenced in this cython-users post: https://groups.google.com/forum/#!topic/cython-users/xYZLKPAwmTE It appears that keyword arguments to functions can't be used as the value for prange(num_threads=...). It's unclear why, but it causes my compiler to crash with "undefined variable __pyx_v_num_threads". I don't have a trac account, but could set one up to file this, if necessary. Ray Jones -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Thu Oct 22 10:36:47 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 22 Oct 2015 01:36:47 -0700 Subject: [Cython] 'Class' redeclared error if C++ class & file name match those of the wrapper class In-Reply-To: References: Message-ID: On Wed, Oct 21, 2015 at 8:52 AM, Nathan Goldbaum wrote: > Hi all, > > I'm forwarding this at the request of David Nemesky ("savior" on > Freenode), who I've been chatting with on the cython IRC channel. He was > having trouble signing up to the cython-devel mailing list, so I've gone > ahead and sent his bug report in for him. His report follows below. > > Apparently this error happens under very specific circumstances, but I > managed to run into them. When wrapping a C++ class, Cython reports a > "'ClassName' redeclared" error, if: > - the name of the .pyx is the same as the name of the .pxd > - the name of the python class is the same as the name of the C++ class > > Mismatch in either the file- or the class name results in a successful > compilation. > Correct, this would result in the two objects having the same name in the same namespace--you can't name the Python class the same as the C++ class any more than you can have two C++ classes named the same thing in the same namespace, or two typedefs named the same thing, etc. The solution is to put it in a separate .pxd file and refer to it either qualified with the module name or using an alias like "from ... cimport CppClass as CppClass_c". You can also do http://docs.cython.org/src/userguide/external_C_code.html#resolving-naming-conflicts-c-name-specifications > I have modified my fork of the cython-cpp-test repo to make the issue > easily reproducible. In https://github.com/DavidNemeskey/cython-cpp-test/, > try `python setup.py build_ext --inplace` (do not forget to delete all .cpp > and .so files between calls) in the following branches: > - cython_bug (diff file, diff class) -> OK > - diff_file_same_class -> OK > - same_file_diff_class -> OK > - same_file_same_class -> ERROR (https://bpaste.net/show/2c1af5fa5ee2) > > Since Cython compiles xxx.pyx to xxx.cpp in this case, in order to avoid > problems with testclass.cpp being overwritten, I implemented the class in > testclass.hpp (in the original repo, there was a .h and a .cpp file). > However, implementing everything in the header is not a pre-requisite of > the bug, as is evident from branch same_file_same_class_diff_c. > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlosjosepita at gmail.com Sun Oct 25 14:20:42 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 25 Oct 2015 15:20:42 -0300 Subject: [Cython] [Bug] Generator comprehension not compiling In-Reply-To: References: Message-ID: Hi, I've been doing some debugging. The problem here is that the cfunc directive forces an AdjustDefByDirectives visit. This is ok for the f node but not for the genexpr DefNode (i.e. node.body.stats[0].expr.args[0].def_node, where node is the f node) which inherits the cfunc directive from f and is also (but wrongly) transformed to a CFuncDefNode. I'm not sure whether there is a valid use case for inheriting cfunc or not. The current implementation just set the visitor directives attribute to the directives in a CompilerDirectivesNode, so the same directives are active for the entire recursive descent. An alternative is to look at node.directives (when available) instead of visitor.directives, this in cases for which directive inheritance is undesirable. Here [1] is a PR that does what I've just described, but only for cfunc. I consider it a dirty hack until more general criteria are discussed and adopted (for example, it seems inconsistent to me doing this for cfunc and not for ccall). Cheers -- Carlos [1] https://github.com/cython/cython/pull/459 On Fri, Oct 16, 2015 at 3:53 PM, Carlos Pita wrote: > Hi, > > in cython 0.24.0a0 this: > > ---------- > import cython as cy > > @cy.cfunc > def f(cats): > ''.join(c for c in cats) > ---------- > > fails to compile with: Compiler crash in AnalyseDeclarationsTransform. > > Nevertheless the following close variations do compile: > > ---------- > import cython as cy > > @cy.cfunc > def f(cats): > ''.join([c for c in cats]) > ---------- > > ---------- > cdef f(cats): > ''.join(c for c in cats) > ---------- > > Cheers > -- > Carlos > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdemeyer at cage.ugent.be Wed Oct 28 12:30:54 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Wed, 28 Oct 2015 17:30:54 +0100 Subject: [Cython] Do we really need realpath()? Message-ID: <5630F83E.7010608@cage.ugent.be> Hello, I am currently profiling Cython's dependency-checking code when compiling the SageMath Cython library. I am skipping the actual cythonization, so it's really just the first "for m in module_list" loop in cythonize(). I noticed that about 10% of total time is spent in os.path.realpath() calls in Cython/Build/Dependencies.py. Do we really need realpath()? Isn't abspath() sufficient? These realpath() calls have been introduced in commit 1bfd57ef7b1a350135024b19a9352411a952c553 Author: R. Andrew Ohana Date: Tue Apr 9 14:03:19 2013 -0700 copy sources and depends instead of headers Jeroen. From robertwb at math.washington.edu Wed Oct 28 13:36:42 2015 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 28 Oct 2015 10:36:42 -0700 Subject: [Cython] Do we really need realpath()? In-Reply-To: <5630F83E.7010608@cage.ugent.be> References: <5630F83E.7010608@cage.ugent.be> Message-ID: I'm not sure why realpath was used here; that does seem quite expensive. On Wed, Oct 28, 2015 at 9:30 AM, Jeroen Demeyer wrote: > Hello, > > I am currently profiling Cython's dependency-checking code when compiling > the SageMath Cython library. I am skipping the actual cythonization, so it's > really just the first "for m in module_list" loop in cythonize(). > > I noticed that about 10% of total time is spent in os.path.realpath() calls > in Cython/Build/Dependencies.py. Do we really need realpath()? Isn't > abspath() sufficient? > > These realpath() calls have been introduced in > > > commit 1bfd57ef7b1a350135024b19a9352411a952c553 > Author: R. Andrew Ohana > Date: Tue Apr 9 14:03:19 2013 -0700 > > copy sources and depends instead of headers > > > Jeroen. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From atash at google.com Wed Oct 28 21:41:24 2015 From: atash at google.com (Masood Malekghassemi) Date: Wed, 28 Oct 2015 18:41:24 -0700 Subject: [Cython] Cython coverage and multiple projects In-Reply-To: References: <56192E73.9030008@behnel.de> Message-ID: After reorganizing the code such that all the tests are located back in the same project, setting up the running of tests in-tree, and using the unittest module directly, there's been much success! Automatically suffixed .coverage files are generated per test case and merged at the end (keeping open the possibility of process-level parallelism later). The coverage report at the end after invoking `coverage combine` on the command line is as expected. There's a strange caveat, though: using the coverage.py API and calling 'coverage.Coverage().combine()' following the test runs erases the Cython coverage data from the .coverage files. I haven't poked at it much yet, as this is only a papercut for us, but if it's an issue with the Cython plug-in, I figure y'all ought to know. Thanks again! -------------- next part -------------- An HTML attachment was scrubbed... URL: