From lists at onerussian.com Wed Aug 1 17:21:30 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Wed, 1 Aug 2012 11:21:30 -0400 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <50182E8C.1060409@behnel.de> References: <500DB7FA.4000409@behnel.de> <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> Message-ID: <20120801152130.GZ5788@onerussian.com> sorry about the delay -- was not monitoring the ML tight enough ;) So it is the commit 8443607d7dffc7c8443d70036e0cce6aaa9c26e2 Author: Stefan Behnel Date: Tue Jul 31 21:49:20 2012 +0200 determine buffer typegroup of integer dtypes based on signedness at C compile time ... I pulled current master and that specific test which failed before passes now: (s390x-sid)yoh at zelenka:~/cython/cython$ git describe --tags 0.16rc1-547-g68811fa (s390x-sid)yoh at zelenka:~/cython/cython$ OPT="-g -O0" /usr/bin/python runtests.py -vv memoryview_compare_type_pointers --no-cleanup Python 2.7.3 (default, Jul 14 2012, 05:19:55) [GCC 4.6.3] Running tests against Cython 0.17.beta1 68811fa9946e4253ad405ba3011512a32807bc7b Backends: c,cpp runTest (__main__.EndToEndTest) End-to-end memoryview_compare_type_pointers ... ok ---------------------------------------------------------------------- Ran 1 test in 8.641s OK ALL DONE Let me run the entire suite and report back if any oddity. Thanks! On Tue, 31 Jul 2012, Stefan Behnel wrote: > Yaroslav, could you give it a try on the Debian build servers? > Stefan > diff -r 0d14a856f2cd Cython/Compiler/Buffer.py > --- a/Cython/Compiler/Buffer.py Tue Jul 31 20:05:37 2012 +0200 > +++ b/Cython/Compiler/Buffer.py Tue Jul 31 21:10:13 2012 +0200 > @@ -680,32 +680,25 @@ > rep = str(dtype) > flags = "0" > - > + is_unsigned = "0" > if dtype.is_int: > - if dtype.signed == 0: -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From lists at onerussian.com Wed Aug 1 18:08:22 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Wed, 1 Aug 2012 12:08:22 -0400 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <20120801152130.GZ5788@onerussian.com> References: <500DB7FA.4000409@behnel.de> <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> Message-ID: <20120801160822.GA5788@onerussian.com> ok, the output of running OPT="-g -O0" /usr/bin/python runtests.py -vv --no-cleanup 2>&1 | tee ../0.16rc1-547-g68811fa-tests-output.txt on a cleaned git repository on an s390x boiling down to Ran 6891 tests in 1098.734s FAILED (failures=42, errors=2) is at http://www.onerussian.com/tmp/0.16rc1-547-g68811fa-tests-output.txt.gz especially interesting are failures like: ValueError: Buffer dtype mismatch, expected 'char' but got 'char' in 'UnpackedStruct.a' ;-) or may be I should have invoked tests anyhow differently (build manually first etc)? On Wed, 01 Aug 2012, Yaroslav Halchenko wrote: > sorry about the delay -- was not monitoring the ML tight enough ;) > So it is the > commit 8443607d7dffc7c8443d70036e0cce6aaa9c26e2 > Author: Stefan Behnel > Date: Tue Jul 31 21:49:20 2012 +0200 > determine buffer typegroup of integer dtypes based on signedness at C compile time > ... > I pulled current master and that specific test which failed before passes now: > (s390x-sid)yoh at zelenka:~/cython/cython$ git describe --tags > 0.16rc1-547-g68811fa > (s390x-sid)yoh at zelenka:~/cython/cython$ OPT="-g -O0" /usr/bin/python runtests.py -vv memoryview_compare_type_pointers --no-cleanup > Python 2.7.3 (default, Jul 14 2012, 05:19:55) > [GCC 4.6.3] > Running tests against Cython 0.17.beta1 68811fa9946e4253ad405ba3011512a32807bc7b > Backends: c,cpp > runTest (__main__.EndToEndTest) > End-to-end memoryview_compare_type_pointers ... ok > ---------------------------------------------------------------------- > Ran 1 test in 8.641s > OK > ALL DONE > Let me run the entire suite and report back if any oddity. Thanks! -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From lists at onerussian.com Wed Aug 1 18:19:48 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Wed, 1 Aug 2012 12:19:48 -0400 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <20120801160822.GA5788@onerussian.com> References: <500DB7FA.4000409@behnel.de> <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> <20120801160822.GA5788@onerussian.com> Message-ID: <20120801161947.GB5788@onerussian.com> minor note -- could someone push the tag (annotated or signed preferably) for 0.17rc1? NB I am not sure what is the status on PEP 386 [1] (not yet adopted afaik) but it might be worthwhile following it and/or existing disutils.version.StrictVersion since having In [2]: Cython.__version__ Out[2]: '0.17.beta1' is not digested well by some depending projects -- there were 2 failures to build against this beta release of Cython in Debian due to difficulty with parsing the version -- bzr and xpra I believe [2]. So may be it is worth making it '0.17b1' (and use corresponding tags accordingly)? [1] http://www.python.org/dev/peps/pep-0386/ [2] http://bugs.debian.org/683133 On Wed, 01 Aug 2012, Yaroslav Halchenko wrote: > ok, the output of running > OPT="-g -O0" /usr/bin/python runtests.py -vv --no-cleanup 2>&1 | tee ../0.16rc1-547-g68811fa-tests-output.txt > on a cleaned git repository on an s390x boiling down to > Ran 6891 tests in 1098.734s > FAILED (failures=42, errors=2) > is at > http://www.onerussian.com/tmp/0.16rc1-547-g68811fa-tests-output.txt.gz > especially interesting are failures like: > ValueError: Buffer dtype mismatch, expected 'char' but got 'char' in 'UnpackedStruct.a' > ;-) > or may be I should have invoked tests anyhow differently (build manually > first etc)? -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From brad.froehle at gmail.com Wed Aug 1 18:35:08 2012 From: brad.froehle at gmail.com (Bradley M. Froehle) Date: Wed, 1 Aug 2012 09:35:08 -0700 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <20120801161947.GB5788@onerussian.com> References: <500DB7FA.4000409@behnel.de> <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> <20120801160822.GA5788@onerussian.com> <20120801161947.GB5788@onerussian.com> Message-ID: Yes, this versioning has also impacted mpi4py which had to add some pretty ugly code in setup.py to work around it: https://code.google.com/p/mpi4py/source/detail?r=841e9df -Brad > NB I am not sure what is the status on PEP 386 [1] (not yet adopted > afaik) but it might be worthwhile following it and/or existing > disutils.version.StrictVersion since having > > In [2]: Cython.__version__ > Out[2]: '0.17.beta1' > > is not digested well by some depending projects -- there were 2 failures to > build against this beta release of Cython in Debian due to difficulty with > parsing the version -- bzr and xpra I believe [2]. So may be it is worth > making it '0.17b1' (and use corresponding tags accordingly)? > > [1] http://www.python.org/dev/peps/pep-0386/ > [2] http://bugs.debian.org/683133 -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Thu Aug 2 00:41:03 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 1 Aug 2012 23:41:03 +0100 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <50182E8C.1060409@behnel.de> References: <500DB7FA.4000409@behnel.de> <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> Message-ID: On 31 July 2012 20:14, Stefan Behnel wrote: > Robert Bradshaw, 31.07.2012 19:24: >> On Tue, Jul 31, 2012 at 7:24 AM, Stefan Behnel wrote: >>> mark florisson, 28.07.2012 14:04: >>>> On 27 July 2012 23:30, Bradley Froehle wrote: >>>>> Thanks to the work of Yaroslav Halchenko, there is an experimental Debian >>>>> package for Cython 0.17.beta1 -- http://packages.qa.debian.org/c/cython.html >>>>> >>>>> However, the builds are showing a lot of test failures on non-amd64 sytems. >>>>> See https://buildd.debian.org/status/package.php?p=cython&suite=experimental >>>>> Many of the errors seem related to assumptions made about the sizes of >>>>> various integers (int vs. long vs long long). Ideally we'll be able to >>>>> clean up these failures before the final release. >>>> >>>> Thanks, I think it's mostly the tests that are wrong. I'll try to get >>>> it fixed on linux 32 bit. >>> >>> I'm not sure the it's only the tests. The "char" vs. "unsigned char" errors >>> hint at platform specific differences, which might trigger bugs (aka. >>> incorrect assumptions) in the generated memory view code. Specifically, >>> plain "char" is unsigned at least on ARM and PowerPC, both of which fail >>> with this error. >> >> Yes, I think somewhere we're assuming char is signed, but wasn't able >> to see where in my quick investigations. > > Yes, it wasn't immediately obvious to me either. Here is a patch that > *might* fix the issue - obviously untested for the platforms in question. > > Yaroslav, could you give it a try on the Debian build servers? > > Stefan > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > Thanks for the fix. I also pushed a fix for one more test numpy_test related to fused types dispatching. That passes all tests for me on 32 bit linux. From lists at onerussian.com Thu Aug 2 16:30:32 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 2 Aug 2012 10:30:32 -0400 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: References: <500DB7FA.4000409@behnel.de> <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> Message-ID: <20120802143031.GC5788@onerussian.com> On Wed, 01 Aug 2012, mark florisson wrote: > Thanks for the fix. I also pushed a fix for one more test numpy_test > related to fused types dispatching. That passes all tests for me on 32 > bit linux. > > Yaroslav, could you give it a try on the Debian build servers? FWIW -- 0.16rc1-550-g8880c78 hasn't added nor fixed any of the failures on s390x in comparison to 0.16rc1-547-g68811fa ;-) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From stefan_ml at behnel.de Thu Aug 2 16:54:51 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 02 Aug 2012 16:54:51 +0200 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: References: <500DB7FA.4000409@behnel.de> <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> <20120801160822.GA5788@onerussian.com> <20120801161947.GB5788@onerussian.com> Message-ID: <501A94BB.8000809@behnel.de> Bradley M. Froehle, 01.08.2012 18:35: > Yes, this versioning has also impacted mpi4py which had to add some pretty ugly code in setup.py to work around it: > > https://code.google.com/p/mpi4py/source/detail?r=841e9df > >> I am not sure what is the status on PEP 386 [1] (not yet adopted afaik) It's in state "Accepted". > but it might be worthwhile following it and/or existing >> disutils.version.StrictVersion since having >> >> In [2]: Cython.__version__ >> Out[2]: '0.17.beta1' >> >> is not digested well by some depending projects -- there were 2 failures to >> build against this beta release of Cython in Debian due to difficulty with >> parsing the version -- bzr and xpra I believe [2]. So may be it is worth >> making it '0.17b1' (and use corresponding tags accordingly)? I guess so. That would fit the StrictVersion scheme. Stefan From lists at onerussian.com Thu Aug 2 19:02:48 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 2 Aug 2012 13:02:48 -0400 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <501A94BB.8000809@behnel.de> References: <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> <20120801160822.GA5788@onerussian.com> <20120801161947.GB5788@onerussian.com> <501A94BB.8000809@behnel.de> Message-ID: <20120802170248.GA5866@onerussian.com> On Thu, 02 Aug 2012, Stefan Behnel wrote: > Bradley M. Froehle, 01.08.2012 18:35: > > Yes, this versioning has also impacted mpi4py which had to add some pretty ugly code in setup.py to work around it: > > https://code.google.com/p/mpi4py/source/detail?r=841e9df > >> I am not sure what is the status on PEP 386 [1] (not yet adopted afaik) > It's in state "Accepted". ah right -- now it is in python3 (but not in python2): (git)novo:~/proj/misc/cpython[2.7]git $> git grep NormalizedVersion 2.7 $> git grep NormalizedVersion master master:Doc/library/packaging.version.rst:.. class:: NormalizedVersion(self, s, error_on_huge_major_num=True) master:Doc/library/packaging.version.rst: >>> NormalizedVersion('1.2b1') < NormalizedVersion('1.2') master:Doc/library/packaging.version.rst: :class:`NormalizedVersion` is used internally by :class:`VersionPredicate` to master:Doc/library/packaging.version.rst: :class:`NormalizedVersion`. master:Doc/library/packaging.version.rst: >>> NormalizedVersion("irrational_version_number") master:Lib/packaging/command/bdist_msi.py:from packaging.version import NormalizedVersion master:Lib/packaging/command/bdist_msi.py:class MSIVersion(NormalizedVersion): master:Lib/packaging/errors.py: See `error_on_huge_major_num` option in `NormalizedVersion` for details. master:Lib/packaging/pypi/dist.py:from packaging.version import (suggest_normalized_version, NormalizedVersion, master:Lib/packaging/pypi/dist.py: self._version = NormalizedVersion(version) master:Lib/packaging/tests/test_version.py:from packaging.version import NormalizedVersion as V master:Lib/packaging/tests/test_version.py: self.assertEqual(repr(V('1.0')), "NormalizedVersion('1.0')") master:Lib/packaging/tests/test_version.py: TypeError: cannot compare NormalizedVersion and str master:Lib/packaging/tests/test_version.py: TypeError: cannot compare NormalizedVersion and str master:Lib/packaging/version.py:__all__ = ['NormalizedVersion', 'suggest_normalized_version', master:Lib/packaging/version.py:class NormalizedVersion: master:Lib/packaging/version.py: """Create a NormalizedVersion instance from a version string. master:Lib/packaging/version.py: the introduction of `NormalizedVersion` was version numbers like master:Lib/packaging/version.py: if not isinstance(other, NormalizedVersion): master:Lib/packaging/version.py: if not isinstance(other, NormalizedVersion): master:Lib/packaging/version.py: If you have a version string that isn't rational (i.e. NormalizedVersion master:Lib/packaging/version.py: - 2312 (53.93%) match NormalizedVersion without change master:Lib/packaging/version.py: NormalizedVersion(s) master:Lib/packaging/version.py: NormalizedVersion(rs) master:Lib/packaging/version.py: return comp, NormalizedVersion(version) master:Lib/packaging/version.py: version = NormalizedVersion(version) > >> parsing the version -- bzr and xpra I believe [2]. So may be it is worth > >> making it '0.17b1' (and use corresponding tags accordingly)? > I guess so. That would fit the StrictVersion scheme. just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree: $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' True at least in python3 it pukes: $> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ c = self._cmp(other) File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp if self.version < other.version: TypeError: unorderable types: tuple() < list() -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From stefan_ml at behnel.de Thu Aug 2 21:35:43 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 02 Aug 2012 21:35:43 +0200 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <20120802170248.GA5866@onerussian.com> References: <55a9d45a-27ea-428f-9fb6-8a1531b2edac@googlegroups.com> <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> <20120801160822.GA5788@onerussian.com> <20120801161947.GB5788@onerussian.com> <501A94BB.8000809@behnel.de> <20120802170248.GA5866@onerussian.com> Message-ID: <501AD68F.3080303@behnel.de> Yaroslav Halchenko, 02.08.2012 19:02: > just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree: > > $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' > True > > at least in python3 it pukes: > > $> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ > c = self._cmp(other) > File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp > if self.version < other.version: > TypeError: unorderable types: tuple() < list() That's surprising at first sight, but I don't think it hurts all that much. People would normally use either of them, not both. Anyway, the next release candidate of Cython will be called 0.17c1. Both the LooseVersion and the NormalizedVersion should be able to handle that. Stefan From lists at onerussian.com Thu Aug 2 21:54:49 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 2 Aug 2012 15:54:49 -0400 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <501AD68F.3080303@behnel.de> References: <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> <20120801160822.GA5788@onerussian.com> <20120801161947.GB5788@onerussian.com> <501A94BB.8000809@behnel.de> <20120802170248.GA5866@onerussian.com> <501AD68F.3080303@behnel.de> Message-ID: <20120802195449.GB5866@onerussian.com> On Thu, 02 Aug 2012, Stefan Behnel wrote: > > just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree: > > $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' > > True > > at least in python3 it pukes: > > $> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' > > Traceback (most recent call last): > > File "", line 1, in > > File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ > > c = self._cmp(other) > > File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp > > if self.version < other.version: > > TypeError: unorderable types: tuple() < list() > That's surprising at first sight, but I don't think it hurts all that much. > People would normally use either of them, not both. I understand that but absence of error in python2.x case (thus suggesting supporting such a comparison) is misleading at least > Anyway, the next release candidate of Cython will be called 0.17c1. Both > the LooseVersion and the NormalizedVersion should be able to handle that. not quite: 1. $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17c1"))' Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/distutils/version.py", line 40, in __init__ self.parse(vstring) File "/usr/lib/python2.7/distutils/version.py", line 107, in parse raise ValueError, "invalid version number '%s'" % vstring ValueError: invalid version number '0.17c1' so it needs to be 0.17b1 I guess $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17b1"))' True 2. LooseVersion is too loose to be used with standardized suffixes, so it would not sort those release candidates appropriately $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(LV("0.17") > LV("0.17b1"))' False -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From dalcinl at gmail.com Fri Aug 3 00:36:55 2012 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 2 Aug 2012 19:36:55 -0300 Subject: [Cython] Bad interaction between cimported types and module cleanup Message-ID: Basically, the module cleanup function nullifies the type ptr of cimported parent types, but tp_dealloc slots of children types access these null pointers, then you get a segfault. Looking at Py_Finalize() , atexit functions are run BEFORE a gargabe collection step and the destroy of all modules. @Stefan, I remember you also had issues with module cleanup and object deallocation. Did you find a solution? Could you give a quick look at the following patch? Do you think the extra pointer deref for getting Py_TYPE(o)->tp_base->tp_dealloc would be a performance regression? BTW, The "if 0" branch is the original code, the else branch is my proposed fix. diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py index 2472de3..8758967 100644 --- a/Cython/Compiler/ModuleNode.py +++ b/Cython/Compiler/ModuleNode.py @@ -1111,7 +1111,12 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode): if base_type: tp_dealloc = TypeSlots.get_base_slot_function(scope, tp_slot) if tp_dealloc is None: - tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname + if 0: + # XXX bad interaction between + # cimported types and module cleanup + tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname + else: + tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc" code.putln( "%s(o);" % tp_dealloc) else: -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From fperez.net at gmail.com Fri Aug 3 01:14:20 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 2 Aug 2012 16:14:20 -0700 Subject: [Cython] Fwd: [ANN] SIAM Conference on Computational Science & Engineering Submission Deadlines Approaching! In-Reply-To: References: Message-ID: Dear Colleagues, the SIAM CSE13 conference will be held next year in Boston, and this is a conference that is well suited for much of the type of work that goes on in the open source scientific Python development community (and Julia). The conference is co-chaired by Hans-Petter Langtangen, well known around these parts for his several books on scientific computing with Python and for having led a campus-wide adoption of Python as the core computational foundation across the University of Oslo. I am also on the program committee, as well as Randy LeVeque and other Python-friendly folks. An excellent way to participate is to organize a one- or two-part minisymposium on a specific topic with a group of related speakers (instructions at http://www.siam.org/meetings/cse13/submissions.php). Please note that the MS deadline is fast approaching: August 13, 2012. If you have any further questions, don't hesitate to contact me or one of the other organizers if you feel they can address your concerns more directly: "Fernando Perez" "Randy LeVeque" (Reproducible research track) "Hans Petter Langtangen" (Conference co-chair) "Karen Willcox" (conference chair) ---------- Forwarded message ---------- From: Karen Willcox Date: Tue, Jul 24, 2012 at 6:08 AM Subject: [SIAM-CSE] SIAM Conference on Computational Science & Engineering Submission Deadlines Approaching! To: SIAM-CSE at siam.org *SIAM Conference on Computational Science & Engineering (CSE13)* February 25-March 1, 2013 The Westin Boston Waterfront, Boston, Massachusetts, USA**** ** ** SUBMISSION DEADLINES ARE APPROACHING!**** August 13, 2012: Minisymposium proposals September 10, 2012: Abstracts for contributed and minisymposium speakers**** Visit http://www.siam.org/meetings/cse13/submissions.php to submit.**** ** ** Twitter hashtag: #SIAMcse13**** ** ** For more information about the conference, visit * http://www.siam.org/meetings/cse13/* or contact SIAM Conference Department at meetings at siam.org.**** -- Karen Willcox Professor and Associate Department Head Department of Aeronautics and Astronautics, MIT http://acdl.mit.edu/willcox.html _______________________________________________ SIAM-CSE mailing list To post messages to the list please send them to: SIAM-CSE at siam.org http://lists.siam.org/mailman/listinfo/siam-cse -------------- next part -------------- An HTML attachment was scrubbed... URL: From mongi3 at gmail.com Fri Aug 3 02:18:05 2012 From: mongi3 at gmail.com (Jeff Copeland) Date: Thu, 2 Aug 2012 17:18:05 -0700 Subject: [Cython] Add includes to generated header files Message-ID: I'm working on some C++ projects and I'm using cython to embed python code. In an effort to make things easy for C++ devs not as familiar with cython/python I've made a small change to insert #includes in generated header files the same as is done in generated c code. This allows one who is using code processed with cython to just include the header and not worry about any other headers that may be required to match data types. Not having these headers is especially problematic because otherwise the error message presented at compilation can sometimes be quite cryptic. Patch is attached. Any reason this should not be done? Thanks, Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Updated-to-include-headers-in-.h-file-when-generated.patch Type: application/octet-stream Size: 921 bytes Desc: not available URL: From stefan_ml at behnel.de Fri Aug 3 05:43:33 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Aug 2012 05:43:33 +0200 Subject: [Cython] Add includes to generated header files In-Reply-To: References: Message-ID: <501B48E5.2000402@behnel.de> Jeff Copeland, 03.08.2012 02:18: > I'm working on some C++ projects and I'm using cython to embed python > code. In an effort to make things easy for C++ devs not as familiar > with cython/python I've made a small change to insert #includes in > generated header files the same as is done in generated c code. This > allows one who is using code processed with cython to just include the > header and not worry about any other headers that may be required to > match data types. > > Not having these headers is especially problematic because otherwise > the error message presented at compilation can sometimes be quite > cryptic. > > Patch is attached. Any reason this should not be done? Hmm, didn't try your patch, but I guess it inserts includes for *all* header files that the module that exports the C-API uses internally, right? If so, I'm not sure that's always wanted. I see the advantage, sure, but it may expose a lot of implementation details that external code may not normally have to care about. On the other hand, if extension types are exported, for example, it's somewhat unlikely that their struct would compile without at least some of those header files, because they almost always wrap some kind of externally defined struct, pointer or whatever kind of other data type. That sounds to me like we should make it a configurable option, something like "capi_reexport_cincludes". Stefan From stefan_ml at behnel.de Fri Aug 3 06:04:55 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Aug 2012 06:04:55 +0200 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: References: Message-ID: <501B4DE7.6060407@behnel.de> Lisandro Dalcin, 03.08.2012 00:36: > Basically, the module cleanup function nullifies the type ptr of > cimported parent types, but tp_dealloc slots of children types access > these null pointers, then you get a segfault. Looking at Py_Finalize() > , atexit functions are run BEFORE a gargabe collection step and the > destroy of all modules. > > @Stefan, I remember you also had issues with module cleanup and object > deallocation. Did you find a solution? Nothing but setting the cleanup code level from 3 down to 2. > Could you give a quick look at the following patch? Looks like it should handle this case, yes. Maybe we could explicitly only apply it to cimported (i.e. non module local) types, but I guess that's what "tp_dealloc is None" already handles. > Do you think the > extra pointer deref for getting Py_TYPE(o)->tp_base->tp_dealloc would > be a performance regression? I have no idea, but I doubt that it would be any significant, given that the call is an indirect one already and that CPython is about to do something with its heap memory management shortly afterwards, with potentially a couple of additional DECREF calls in between. All of this should be costly enough to make the difference negligible. > diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py > index 2472de3..8758967 100644 > --- a/Cython/Compiler/ModuleNode.py > +++ b/Cython/Compiler/ModuleNode.py > @@ -1111,7 +1111,12 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode): > if base_type: > tp_dealloc = TypeSlots.get_base_slot_function(scope, tp_slot) > if tp_dealloc is None: > - tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname > + if 0: > + # XXX bad interaction between > + # cimported types and module cleanup > + tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname > + else: > + tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc" > code.putln( > "%s(o);" % tp_dealloc) > else: Any other opinions on this? Lisandro, could you open a pull request with only the single line change and a comment that properly explains why we are doing this? Stefan From stefan_ml at behnel.de Fri Aug 3 06:07:46 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Aug 2012 06:07:46 +0200 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: <501B4DE7.6060407@behnel.de> References: <501B4DE7.6060407@behnel.de> Message-ID: <501B4E92.5050200@behnel.de> Stefan Behnel, 03.08.2012 06:04: > Lisandro, could you open a pull request with only the single line change > and a comment that properly explains why we are doing this? A code comment, just to be clear. Stefan From mongi3 at gmail.com Fri Aug 3 06:11:22 2012 From: mongi3 at gmail.com (Jeff Copeland) Date: Thu, 2 Aug 2012 21:11:22 -0700 Subject: [Cython] Add includes to generated header files In-Reply-To: <501B48E5.2000402@behnel.de> References: <501B48E5.2000402@behnel.de> Message-ID: On Thu, Aug 2, 2012 at 8:43 PM, Stefan Behnel wrote: > Jeff Copeland, 03.08.2012 02:18: >> I'm working on some C++ projects and I'm using cython to embed python >> code. In an effort to make things easy for C++ devs not as familiar >> with cython/python I've made a small change to insert #includes in >> generated header files the same as is done in generated c code. This >> allows one who is using code processed with cython to just include the >> header and not worry about any other headers that may be required to >> match data types. >> >> Not having these headers is especially problematic because otherwise >> the error message presented at compilation can sometimes be quite >> cryptic. >> >> Patch is attached. Any reason this should not be done? > > Hmm, didn't try your patch, but I guess it inserts includes for *all* > header files that the module that exports the C-API uses internally, right? Yes, you've got it. Certainly quick and dirty to satisfy my immediate need. > If so, I'm not sure that's always wanted. I see the advantage, sure, but it > may expose a lot of implementation details that external code may not > normally have to care about. > > On the other hand, if extension types are exported, for example, it's > somewhat unlikely that their struct would compile without at least some of > those header files, because they almost always wrap some kind of externally > defined struct, pointer or whatever kind of other data type. Exactly. > That sounds to me like we should make it a configurable option, something > like "capi_reexport_cincludes". I like that thought. The other thing I'd though of is perhaps some syntax in the pyx itself to designate an include file as one that should show in a generated header. When I was looking for a solution to the issue that's what I started looking for in the documents but of course came up empty. I figured it may be something like the case for inclusion in the c file but with a public/api keyword thrown in: cdef extern public from "spam.h": pass Syntax change would probably be more involved. I'll have a go at doing a patch with the option added. Do you mean as a command-line option to cython? Jeff From stefan_ml at behnel.de Fri Aug 3 06:18:27 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Aug 2012 06:18:27 +0200 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 1 released In-Reply-To: <20120802195449.GB5866@onerussian.com> References: <5017EA9C.8030803@behnel.de> <50182E8C.1060409@behnel.de> <20120801152130.GZ5788@onerussian.com> <20120801160822.GA5788@onerussian.com> <20120801161947.GB5788@onerussian.com> <501A94BB.8000809@behnel.de> <20120802170248.GA5866@onerussian.com> <501AD68F.3080303@behnel.de> <20120802195449.GB5866@onerussian.com> Message-ID: <501B5113.20003@behnel.de> Yaroslav Halchenko, 02.08.2012 21:54: > On Thu, 02 Aug 2012, Stefan Behnel wrote: >>> just a side-note -- I didn't know that StrictVersion doesn't play nicely with LooseVersion to any degree: > >>> $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print SV("0.17") > LV("0.18")' >>> True > >>> at least in python3 it pukes: > >>> $> python3 -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > LV("0.18"))' >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "/usr/lib/python3.2/distutils/version.py", line 70, in __gt__ >>> c = self._cmp(other) >>> File "/usr/lib/python3.2/distutils/version.py", line 179, in _cmp >>> if self.version < other.version: >>> TypeError: unorderable types: tuple() < list() > >> That's surprising at first sight, but I don't think it hurts all that much. >> People would normally use either of them, not both. > > I understand that but absence of error in python2.x case (thus suggesting > supporting such a comparison) is misleading at least > >> Anyway, the next release candidate of Cython will be called 0.17c1. Both >> the LooseVersion and the NormalizedVersion should be able to handle that. > > not quite: > > 1. > > $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17c1"))' > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python2.7/distutils/version.py", line 40, in __init__ > self.parse(vstring) > File "/usr/lib/python2.7/distutils/version.py", line 107, in parse > raise ValueError, "invalid version number '%s'" % vstring > ValueError: invalid version number '0.17c1' > > > so it needs to be 0.17b1 I guess > > $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(SV("0.17") > SV("0.17b1"))' > True > > > 2. LooseVersion is too loose to be used with standardized suffixes, so it > would not sort those release candidates appropriately > > $> python -c 'from distutils.version import LooseVersion as LV, StrictVersion as SV; print(LV("0.17") > LV("0.17b1"))' > False Great. Then we'll have a "release candidate" called "beta" for the sake of backwards compatibility. I'm not entirely happy about that. Stefan From stefan_ml at behnel.de Fri Aug 3 06:52:44 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Aug 2012 06:52:44 +0200 Subject: [Cython] help getting NumPy into a 32-bit Jenkins build environment In-Reply-To: References: <5013B2DF.4020309@behnel.de> Message-ID: <501B591C.2030502@behnel.de> Lisandro Dalcin, 31.07.2012 18:59: > On 28 July 2012 06:37, Stefan Behnel wrote: >> I've been trying to set up a 32bit build environment on Jenkins. Most of it >> works nicely by just switching to "gcc -m32", but NumPy fails to build with >> that setup due to the lack of dependencies. It actually finds them when >> looking for their header files in the wrong (64bit) places, and then fails >> to build when it can't find the expected 32bit libraries. >> >> Is there a way to tell NumPy that I really don't care about BLAS, LAPACK >> and all of its friends and just want bare arrays? Or is there an easy way >> to install 32 bit versions of those dependencies on sage.math? I'd be fine >> with having them built and installed into a local directory (namely, in >> standard directory layout under /home/scoder/jenkins/Tools/libs32/). >> > >>From doc/source/user/install.rst in numpy-dev: > > > Disabling ATLAS and other accelerated libraries > ----------------------------------------------- > > Usage of ATLAS and other accelerated libraries in Numpy can be disabled > via:: > > BLAS=None LAPACK=None ATLAS=None python setup.py build Cool, looks like that did the trick. https://sage.math.washington.edu:8091/hudson/job/py27m-ext-hg/ARCH=m32/12/ Thanks! Stefan From dalcinl at gmail.com Fri Aug 3 18:51:47 2012 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 3 Aug 2012 13:51:47 -0300 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: <501B4E92.5050200@behnel.de> References: <501B4DE7.6060407@behnel.de> <501B4E92.5050200@behnel.de> Message-ID: On 3 August 2012 01:07, Stefan Behnel wrote: > Stefan Behnel, 03.08.2012 06:04: >> Lisandro, could you open a pull request with only the single line change >> and a comment that properly explains why we are doing this? > > A code comment, just to be clear. > Is the comment blow clear enough for you? Stefan, I've never used github seriously, and making a pull request will require time I really do not have. The patch is really trivial, any chance to push it directly? There is a policy is to use pull requests for every change (sorry, I've not been following Cython development closely)? diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py index 2472de3..255b047 100644 --- a/Cython/Compiler/ModuleNode.py +++ b/Cython/Compiler/ModuleNode.py @@ -1111,7 +1111,11 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode): if base_type: tp_dealloc = TypeSlots.get_base_slot_function(scope, tp_slot) if tp_dealloc is None: - tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname + # Using the cimported base type pointer interacts + # badly with module cleanup nullyfing these pointers. + # Use instead the base type pointer from the + # instance's type pointer. + tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc" code.putln( "%s(o);" % tp_dealloc) else: -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Sat Aug 4 08:52:02 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Aug 2012 08:52:02 +0200 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: References: <501B4DE7.6060407@behnel.de> <501B4E92.5050200@behnel.de> Message-ID: <501CC692.40303@behnel.de> Lisandro Dalcin, 03.08.2012 18:51: >> Stefan Behnel, 03.08.2012 06:04: >>> Lisandro, could you open a pull request with only the single line change >>> and a comment that properly explains why we are doing this? > > Is the comment blow clear enough for you? Thanks. I fixed it up a little. :) > Stefan, I've never used github seriously, and making a pull request > will require time I really do not have. github is really cool here, you can basically edit a source file in place and then send it in as a pull request. > The patch is really trivial, > any chance to push it directly? https://github.com/cython/cython/commit/fd6380ab37d32ec67d5cc818c93b8d317f139e4d > There is a policy is to use pull > requests for every change (sorry, I've not been following Cython > development closely)? Sort-of. It makes it easy to see in the history who wrote a patch and who applied it, usually after reviewing it. It's more meant to give proper credit to people than to force them into bureaucratics, though. Stefan From stefan_ml at behnel.de Sat Aug 4 14:59:38 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Aug 2012 14:59:38 +0200 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: References: <501B4DE7.6060407@behnel.de> <501B4E92.5050200@behnel.de> Message-ID: <501D1CBA.6000306@behnel.de> Lisandro Dalcin, 03.08.2012 18:51: > diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py > index 2472de3..255b047 100644 > --- a/Cython/Compiler/ModuleNode.py > +++ b/Cython/Compiler/ModuleNode.py > @@ -1111,7 +1111,11 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode): > if base_type: > tp_dealloc = TypeSlots.get_base_slot_function(scope, tp_slot) > if tp_dealloc is None: > - tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname > + # Using the cimported base type pointer interacts > + # badly with module cleanup nullyfing these pointers. > + # Use instead the base type pointer from the > + # instance's type pointer. > + tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc" > code.putln( > "%s(o);" % tp_dealloc) > else: Tried it, doesn't work. The problem is that this always goes through the actual type of the object, ignoring the type hierarchy completely, which kills the tp_dealloc() call chain and runs into an infinite loop starting from the second inheritance level (or a crash because of multiple DECREF calls on the same fields). Stefan From stefan_ml at behnel.de Sat Aug 4 22:50:26 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Aug 2012 22:50:26 +0200 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: <501D1CBA.6000306@behnel.de> References: <501B4DE7.6060407@behnel.de> <501B4E92.5050200@behnel.de> <501D1CBA.6000306@behnel.de> Message-ID: <501D8B12.1000002@behnel.de> Stefan Behnel, 04.08.2012 14:59: > Lisandro Dalcin, 03.08.2012 18:51: >> diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py >> index 2472de3..255b047 100644 >> --- a/Cython/Compiler/ModuleNode.py >> +++ b/Cython/Compiler/ModuleNode.py >> @@ -1111,7 +1111,11 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode): >> if base_type: >> tp_dealloc = TypeSlots.get_base_slot_function(scope, tp_slot) >> if tp_dealloc is None: >> - tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname >> + # Using the cimported base type pointer interacts >> + # badly with module cleanup nullyfing these pointers. >> + # Use instead the base type pointer from the >> + # instance's type pointer. >> + tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc" >> code.putln( >> "%s(o);" % tp_dealloc) >> else: > > Tried it, doesn't work. The problem is that this always goes through the > actual type of the object, ignoring the type hierarchy completely, which > kills the tp_dealloc() call chain and runs into an infinite loop starting > from the second inheritance level (or a crash because of multiple DECREF > calls on the same fields). Here is a fix that should handle this correctly. https://github.com/cython/cython/commit/a8393fa58741c9ae0647e8fdec5fee4ffd91ddf9 Basically, it checks the type pointer of cimported types on tp_traverse(), tp_clear() and tp_dealloc() and in the unlikely case that they are NULL, it walks the type hierarchy up to the point where it finds the current function and then calls the one in the base type. It is somewhat involved, but I still doubt that it would lead to a real slow down. Maybe tp_traverse() is the one that's a bit more time critical than the others because it can be called more often, but the one additional NULL check in the normal case still shouldn't hurt all that much. It fixes the case in lxml for me, but I didn't shrink that down to a test case yet. Lisandro, if you find the time, it would be nice if you could write one up for the problem you ran into. Stefan From dalcinl at gmail.com Sat Aug 4 23:33:57 2012 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 4 Aug 2012 18:33:57 -0300 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: <501D8B12.1000002@behnel.de> References: <501B4DE7.6060407@behnel.de> <501B4E92.5050200@behnel.de> <501D1CBA.6000306@behnel.de> <501D8B12.1000002@behnel.de> Message-ID: On 4 August 2012 17:50, Stefan Behnel wrote: > Stefan Behnel, 04.08.2012 14:59: >> Lisandro Dalcin, 03.08.2012 18:51: >>> diff --git a/Cython/Compiler/ModuleNode.py b/Cython/Compiler/ModuleNode.py >>> index 2472de3..255b047 100644 >>> --- a/Cython/Compiler/ModuleNode.py >>> +++ b/Cython/Compiler/ModuleNode.py >>> @@ -1111,7 +1111,11 @@ class ModuleNode(Nodes.Node, Nodes.BlockNode): >>> if base_type: >>> tp_dealloc = TypeSlots.get_base_slot_function(scope, tp_slot) >>> if tp_dealloc is None: >>> - tp_dealloc = "%s->tp_dealloc" % base_type.typeptr_cname >>> + # Using the cimported base type pointer interacts >>> + # badly with module cleanup nullyfing these pointers. >>> + # Use instead the base type pointer from the >>> + # instance's type pointer. >>> + tp_dealloc = "Py_TYPE(o)->tp_base->tp_dealloc" >>> code.putln( >>> "%s(o);" % tp_dealloc) >>> else: >> >> Tried it, doesn't work. The problem is that this always goes through the >> actual type of the object, ignoring the type hierarchy completely, which >> kills the tp_dealloc() call chain and runs into an infinite loop starting >> from the second inheritance level (or a crash because of multiple DECREF >> calls on the same fields). > > Here is a fix that should handle this correctly. > > https://github.com/cython/cython/commit/a8393fa58741c9ae0647e8fdec5fee4ffd91ddf9 > > Basically, it checks the type pointer of cimported types on tp_traverse(), > tp_clear() and tp_dealloc() and in the unlikely case that they are NULL, it > walks the type hierarchy up to the point where it finds the current > function and then calls the one in the base type. > Nice! > It is somewhat involved, but I still doubt that it would lead to a real > slow down. Maybe tp_traverse() is the one that's a bit more time critical > than the others because it can be called more often, but the one additional > NULL check in the normal case still shouldn't hurt all that much. > OK. > It fixes the case in lxml for me, but I didn't shrink that down to a test > case yet. Lisandro, if you find the time, it would be nice if you could > write one up for the problem you ran into. > To properly test this we would need to namespace the __cleanup functions registered with atexit, I mean to name them '____cleanup', that way we could loop over registered functions to and call them "by hand". I you agree, I can take care of this and write some tests for module cleanup. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Sat Aug 4 23:45:35 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Aug 2012 23:45:35 +0200 Subject: [Cython] Bad interaction between cimported types and module cleanup In-Reply-To: References: <501B4DE7.6060407@behnel.de> <501B4E92.5050200@behnel.de> <501D1CBA.6000306@behnel.de> <501D8B12.1000002@behnel.de> Message-ID: <501D97FF.4090005@behnel.de> Lisandro Dalcin, 04.08.2012 23:33: > To properly test this we would need to namespace the __cleanup > functions registered with atexit, I mean to name them > '____cleanup' Sure, why not. > that way we could loop over registered > functions to and call them "by hand". I'm not sure if that'll work - wouldn't they still be called another time at exit? IMHO worth trying. The cleanup code should be safe enough to be reentrant, but who knows. If it's not, it might be worth making it safe enough. Then there's also the question *when* to call the cleanup function of a test module. I doubt that it's a good idea to call it from inside the module itself, but calling it from the test runner might work. > I you agree, I can take care of > this and write some tests for module cleanup. Please do. Stefan From stefan_ml at behnel.de Mon Aug 6 23:58:03 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Aug 2012 23:58:03 +0200 Subject: [Cython] remaining open issues for 0.17 Message-ID: <50203DEB.9080108@behnel.de> Hi, please correct me if I'm missing something, but besides the fused types default value thing, I currently see two open issues for 0.17. One is the 32bit problem we have in the buffer code, the other one is the "vile hack" that Robert added as a type inference fix. I just ran into the following code (while compiling Django) which currently crashes the compiler, and I think it's due to one of those "type inference modifies state" issues. """ def func(**kwargs): """ >>> sorted(func(a=3, b=4)) [1, 2, 3, 4] """ return [ arg for arg in [1,2] + kwargs.values() ] """ How should we deal with these? Personally, I consider general C code portability issues more annoying than code that fails to compile but that can be worked around by users. So, unless we can fix the default values and/or the type inference issues soonishly, I would concentrate on the 32bit issues for now and release 0.17 when we get them resolved. Thoughts? Stefan From stefan_ml at behnel.de Tue Aug 7 09:34:58 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 07 Aug 2012 09:34:58 +0200 Subject: [Cython] Anyone for a doc cleanup before the release? Message-ID: <5020C522.1090900@behnel.de> Hi, building the documentation currently yields a number of Sphinx warnings: https://sage.math.washington.edu:8091/hudson/job/cython-docs/lastSuccessfulBuild/warnings1Result/ It would be nice if someone found the time to clean some of them up. Stefan From markflorisson88 at gmail.com Tue Aug 7 11:09:11 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 7 Aug 2012 10:09:11 +0100 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <50203DEB.9080108@behnel.de> References: <50203DEB.9080108@behnel.de> Message-ID: On 6 August 2012 22:58, Stefan Behnel wrote: > Hi, > > please correct me if I'm missing something, but besides the fused types > default value thing, I currently see two open issues for 0.17. One is the > 32bit problem we have in the buffer code, the other one is the "vile hack" > that Robert added as a type inference fix. I just ran into the following > code (while compiling Django) which currently crashes the compiler, and I > think it's due to one of those "type inference modifies state" issues. > > """ > def func(**kwargs): > """ > >>> sorted(func(a=3, b=4)) > [1, 2, 3, 4] > """ > return [ arg for arg in [1,2] + kwargs.values() ] > """ > > How should we deal with these? Personally, I consider general C code > portability issues more annoying than code that fails to compile but that > can be worked around by users. So, unless we can fix the default values > and/or the type inference issues soonishly, I would concentrate on the > 32bit issues for now and release 0.17 when we get them resolved. > > Thoughts? > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel I thought the 32 bit issue was resolved? You pushed a fix and I fixed some tests, so it passed for me. I can run it again to check... From stefan_ml at behnel.de Tue Aug 7 11:53:40 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 07 Aug 2012 11:53:40 +0200 Subject: [Cython] automatically raise MemoryError on exceptional C return values Message-ID: <5020E5A4.8070208@behnel.de> Hi, given how ubiquitous manual memory management is in C, I think it would be nice to let Cython generate the exception raising also for C, not only for C++. The difference is this: cdef extern from "...": char* make_new_buffer() except NULL as MemoryError int append_to_buffer(char* buffer, char* value) \ except -1 as MemoryError c_buffer = make_new_buffer() # raises MemoryError on NULL append_to_buffer(c_buffer, "testdata") # raises MemoryError on -1 append_to_buffer(c_buffer, "moredata") # raises MemoryError on -1 versus this: cdef extern from "...": char* make_new_buffer() int append_to_buffer(char* buffer, char* value) c_buffer = make_new_buffer() if c_buffer is NULL: raise MemoryError() if append_to_buffer(c_buffer, "testdata") == -1: raise MemoryError() if append_to_buffer(c_buffer, "moredata") == -1: raise MemoryError() I don't think it's necessary to support this for anything but MemoryError, since you'd normally want a formatted error message for everything else. Alternative colours for the bike shed: char* make_new_buffer() except NULL raise MemoryError char* make_new_buffer() raise MemoryError for NULL char* make_new_buffer() raise MemoryError from NULL What do you think? Stefan From markflorisson88 at gmail.com Tue Aug 7 12:33:35 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 7 Aug 2012 11:33:35 +0100 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: <5020E5A4.8070208@behnel.de> References: <5020E5A4.8070208@behnel.de> Message-ID: On 7 August 2012 10:53, Stefan Behnel wrote: > Hi, > > given how ubiquitous manual memory management is in C, I think it would be > nice to let Cython generate the exception raising also for C, not only for > C++. The difference is this: > > cdef extern from "...": > char* make_new_buffer() except NULL as MemoryError > int append_to_buffer(char* buffer, char* value) \ > except -1 as MemoryError > > c_buffer = make_new_buffer() # raises MemoryError on NULL > append_to_buffer(c_buffer, "testdata") # raises MemoryError on -1 > append_to_buffer(c_buffer, "moredata") # raises MemoryError on -1 > > versus this: > > cdef extern from "...": > char* make_new_buffer() > int append_to_buffer(char* buffer, char* value) > > c_buffer = make_new_buffer() > if c_buffer is NULL: > raise MemoryError() > if append_to_buffer(c_buffer, "testdata") == -1: > raise MemoryError() > if append_to_buffer(c_buffer, "moredata") == -1: > raise MemoryError() > > I don't think it's necessary to support this for anything but MemoryError, > since you'd normally want a formatted error message for everything else. > > Alternative colours for the bike shed: > > char* make_new_buffer() except NULL raise MemoryError > > char* make_new_buffer() raise MemoryError for NULL > > char* make_new_buffer() raise MemoryError from NULL > > What do you think? > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel I'd say write a wrapper function that does the check, or use an object that encapsulates the buffer allocation and functions. I'm not sure we'd want extra syntax for this. Calling functions is already complicated, this will complicate it even further. From stefan_ml at behnel.de Tue Aug 7 12:44:46 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 07 Aug 2012 12:44:46 +0200 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: References: <5020E5A4.8070208@behnel.de> Message-ID: <5020F19E.1050804@behnel.de> mark florisson, 07.08.2012 12:33: > I'd say write a wrapper function that does the check, or use an object > that encapsulates the buffer allocation and functions. I'm not sure > we'd want extra syntax for this. Calling functions is already > complicated, this will complicate it even further. Funny to see this written by someone who bloats the entire type system with buffers, memory views and slice types, just to keep people from calculating index offsets into arrays. Stefan ;o) From markflorisson88 at gmail.com Tue Aug 7 12:59:04 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 7 Aug 2012 11:59:04 +0100 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: <5020F19E.1050804@behnel.de> References: <5020E5A4.8070208@behnel.de> <5020F19E.1050804@behnel.de> Message-ID: On 7 August 2012 11:44, Stefan Behnel wrote: > mark florisson, 07.08.2012 12:33: >> I'd say write a wrapper function that does the check, or use an object >> that encapsulates the buffer allocation and functions. I'm not sure >> we'd want extra syntax for this. Calling functions is already >> complicated, this will complicate it even further. > > Funny to see this written by someone who bloats the entire type system with > buffers, memory views and slice types, just to keep people from calculating > index offsets into arrays. The only type that was added was the memoryview type, which is supposed to *replace* the buffer type. And it does much more than index calculation. So... yeah. Let's not make a technical discussion personal. > Stefan ;o) > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Tue Aug 7 13:00:04 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 7 Aug 2012 12:00:04 +0100 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: References: <5020E5A4.8070208@behnel.de> <5020F19E.1050804@behnel.de> Message-ID: On 7 August 2012 11:59, mark florisson wrote: > On 7 August 2012 11:44, Stefan Behnel wrote: >> mark florisson, 07.08.2012 12:33: >>> I'd say write a wrapper function that does the check, or use an object >>> that encapsulates the buffer allocation and functions. I'm not sure >>> we'd want extra syntax for this. Calling functions is already >>> complicated, this will complicate it even further. >> >> Funny to see this written by someone who bloats the entire type system with >> buffers, memory views and slice types, just to keep people from calculating >> index offsets into arrays. > > The only type that was added was the memoryview type, which is > supposed to *replace* the buffer type. And it does much more than > index calculation. So... yeah. Let's not make a technical discussion > personal. Besides, feedback seems to suggest people are happy with this "bloat". >> Stefan ;o) >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Tue Aug 7 13:22:24 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 07 Aug 2012 13:22:24 +0200 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: References: <5020E5A4.8070208@behnel.de> <5020F19E.1050804@behnel.de> Message-ID: <5020FA70.7070008@behnel.de> mark florisson, 07.08.2012 13:00: > On 7 August 2012 11:59, mark florisson wrote: >> On 7 August 2012 11:44, Stefan Behnel wrote: >>> mark florisson, 07.08.2012 12:33: >>>> I'd say write a wrapper function that does the check, or use an object >>>> that encapsulates the buffer allocation and functions. I'm not sure >>>> we'd want extra syntax for this. Calling functions is already >>>> complicated, this will complicate it even further. >>> >>> Funny to see this written by someone who bloats the entire type system with >>> buffers, memory views and slice types, just to keep people from calculating >>> index offsets into arrays. >> >> The only type that was added was the memoryview type, which is >> supposed to *replace* the buffer type. And it does much more than >> index calculation. So... yeah. Let's not make a technical discussion >> personal. > > Besides, feedback seems to suggest people are happy with this "bloat". Sorry if you took my comment any serious. It wasn't meant that way. I was only sarcastically suggesting that there are different kinds of users, and to those who don't care about one of the bigger features (in terms of code), it will look like serious code bloat for no good reason. The whole buffer related code is just too excellent an example for that to let the opportunity pass silently. C++ is just another, and I'm sure there are other areas that add a lot of code to the compiler that a substantial subset of our users won't trigger in their whole life. And I'm not saying that's a bad thing. >>> Stefan ;o) Stefan ^^^ From stefan_ml at behnel.de Wed Aug 8 11:14:57 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 08 Aug 2012 11:14:57 +0200 Subject: [Cython] Cython 0.17 beta 2 released Message-ID: <50222E11.90001@behnel.de> Hello everyone, on behalf of the Cython project team, I'm proud to announce the release of our second beta of Cython 0.17. This is another major step forward in the development of the language that will make life easier for a lot of users, rounds up some rough edges of the compiler and adds (preliminary) support for CPython 3.3 and PyPy. Please give this beta release as much testing as you can, so that we can quickly advance towards the final release. Download: http://cython.org/release/Cython-0.17b2.tar.gz Release notes: http://wiki.cython.org/ReleaseNotes-0.17 Documentation: http://docs.cython.org/ Major features of this release include: * vastly improved integration with the C++ STL containers http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library http://docs.cython.org/src/tutorial/strings.html#c-strings * "yield from" delegation between generators (PEP 380) http://www.python.org/dev/peps/pep-0380/ * alpha quality support for PyPy (via cpyext) http://docs.cython.org/src/tutorial/pypy.html Several other features and improvements are listed in the release notes: http://wiki.cython.org/ReleaseNotes-0.17 Have fun, Stefan From robertwb at gmail.com Thu Aug 9 01:45:33 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 8 Aug 2012 16:45:33 -0700 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: <5020E5A4.8070208@behnel.de> References: <5020E5A4.8070208@behnel.de> Message-ID: On Tue, Aug 7, 2012 at 2:53 AM, Stefan Behnel wrote: > Hi, > > given how ubiquitous manual memory management is in C, I think it would be > nice to let Cython generate the exception raising also for C, not only for > C++. The difference is this: > > cdef extern from "...": > char* make_new_buffer() except NULL as MemoryError > int append_to_buffer(char* buffer, char* value) \ > except -1 as MemoryError > > c_buffer = make_new_buffer() # raises MemoryError on NULL > append_to_buffer(c_buffer, "testdata") # raises MemoryError on -1 > append_to_buffer(c_buffer, "moredata") # raises MemoryError on -1 > > versus this: > > cdef extern from "...": > char* make_new_buffer() > int append_to_buffer(char* buffer, char* value) > > c_buffer = make_new_buffer() > if c_buffer is NULL: > raise MemoryError() > if append_to_buffer(c_buffer, "testdata") == -1: > raise MemoryError() > if append_to_buffer(c_buffer, "moredata") == -1: > raise MemoryError() > > I don't think it's necessary to support this for anything but MemoryError, > since you'd normally want a formatted error message for everything else. > > Alternative colours for the bike shed: > > char* make_new_buffer() except NULL raise MemoryError > > char* make_new_buffer() raise MemoryError for NULL > > char* make_new_buffer() raise MemoryError from NULL > > What do you think? I'm +0.5 (with a preference to the "except NULL as MemoryError" flavor). Basically, it is new syntax to make up for the lack of proper meta programming. Alternatively (just throwing it out there, not sure if it's a good idea), could this be done as a decorator of some sort? That's how one would write such a concept in Python. The need to fix types does constrain things. - Robert From dtcaciuc at gmail.com Thu Aug 9 05:40:20 2012 From: dtcaciuc at gmail.com (Dimitri Tcaciuc) Date: Wed, 8 Aug 2012 20:40:20 -0700 Subject: [Cython] Anyone for a doc cleanup before the release? In-Reply-To: <5020C522.1090900@behnel.de> References: <5020C522.1090900@behnel.de> Message-ID: Here's my go at it: https://github.com/cython/cython/pull/141 Cheers! Dimitri. On Tue, Aug 7, 2012 at 12:34 AM, Stefan Behnel wrote: > Hi, > > building the documentation currently yields a number of Sphinx warnings: > > https://sage.math.washington.edu:8091/hudson/job/cython-docs/lastSuccessfulBuild/warnings1Result/ > > It would be nice if someone found the time to clean some of them up. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Thu Aug 9 07:12:12 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Aug 2012 07:12:12 +0200 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: References: <5020E5A4.8070208@behnel.de> Message-ID: <502346AC.1060605@behnel.de> Robert Bradshaw, 09.08.2012 01:45: > On Tue, Aug 7, 2012 at 2:53 AM, Stefan Behnel wrote: >> given how ubiquitous manual memory management is in C, I think it would be >> nice to let Cython generate the exception raising also for C, not only for >> C++. The difference is this: >> >> cdef extern from "...": >> char* make_new_buffer() except NULL as MemoryError >> int append_to_buffer(char* buffer, char* value) \ >> except -1 as MemoryError >> >> c_buffer = make_new_buffer() # raises MemoryError on NULL >> append_to_buffer(c_buffer, "testdata") # raises MemoryError on -1 >> append_to_buffer(c_buffer, "moredata") # raises MemoryError on -1 >> >> versus this: >> >> cdef extern from "...": >> char* make_new_buffer() >> int append_to_buffer(char* buffer, char* value) >> >> c_buffer = make_new_buffer() >> if c_buffer is NULL: >> raise MemoryError() >> if append_to_buffer(c_buffer, "testdata") == -1: >> raise MemoryError() >> if append_to_buffer(c_buffer, "moredata") == -1: >> raise MemoryError() >> >> I don't think it's necessary to support this for anything but MemoryError, >> since you'd normally want a formatted error message for everything else. >> >> Alternative colours for the bike shed: >> >> char* make_new_buffer() except NULL raise MemoryError >> >> char* make_new_buffer() raise MemoryError for NULL >> >> char* make_new_buffer() raise MemoryError from NULL >> >> What do you think? > > I'm +0.5 (with a preference to the "except NULL as MemoryError" > flavor). Basically, it is new syntax to make up for the lack of proper > meta programming. In a way, yes. > Alternatively (just throwing it out there, not sure > if it's a good idea), could this be done as a decorator of some sort? > That's how one would write such a concept in Python. The need to fix > types does constrain things. A decorator on the external function declaration, you mean? Something like this? @autoraise(MemoryError) char* make_new_buffer() except NULL Or like this? @autoraise(MemoryError, error_value="NULL") char* make_new_buffer() I wouldn't mind. Although there isn't currently a pure Python syntax for external declarations that would benefit from this ... Or did you mean that it should be a user implemented decorator? That would be nice, but even fused types would make a simple case like this rather code heavy, I guess. Stefan From robertwb at gmail.com Thu Aug 9 09:15:18 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 9 Aug 2012 00:15:18 -0700 Subject: [Cython] automatically raise MemoryError on exceptional C return values In-Reply-To: <502346AC.1060605@behnel.de> References: <5020E5A4.8070208@behnel.de> <502346AC.1060605@behnel.de> Message-ID: On Wed, Aug 8, 2012 at 10:12 PM, Stefan Behnel wrote: > Robert Bradshaw, 09.08.2012 01:45: >> On Tue, Aug 7, 2012 at 2:53 AM, Stefan Behnel wrote: >>> given how ubiquitous manual memory management is in C, I think it would be >>> nice to let Cython generate the exception raising also for C, not only for >>> C++. The difference is this: >>> >>> cdef extern from "...": >>> char* make_new_buffer() except NULL as MemoryError >>> int append_to_buffer(char* buffer, char* value) \ >>> except -1 as MemoryError >>> >>> c_buffer = make_new_buffer() # raises MemoryError on NULL >>> append_to_buffer(c_buffer, "testdata") # raises MemoryError on -1 >>> append_to_buffer(c_buffer, "moredata") # raises MemoryError on -1 >>> >>> versus this: >>> >>> cdef extern from "...": >>> char* make_new_buffer() >>> int append_to_buffer(char* buffer, char* value) >>> >>> c_buffer = make_new_buffer() >>> if c_buffer is NULL: >>> raise MemoryError() >>> if append_to_buffer(c_buffer, "testdata") == -1: >>> raise MemoryError() >>> if append_to_buffer(c_buffer, "moredata") == -1: >>> raise MemoryError() >>> >>> I don't think it's necessary to support this for anything but MemoryError, >>> since you'd normally want a formatted error message for everything else. >>> >>> Alternative colours for the bike shed: >>> >>> char* make_new_buffer() except NULL raise MemoryError >>> >>> char* make_new_buffer() raise MemoryError for NULL >>> >>> char* make_new_buffer() raise MemoryError from NULL >>> >>> What do you think? >> >> I'm +0.5 (with a preference to the "except NULL as MemoryError" >> flavor). Basically, it is new syntax to make up for the lack of proper >> meta programming. > > In a way, yes. > > >> Alternatively (just throwing it out there, not sure >> if it's a good idea), could this be done as a decorator of some sort? >> That's how one would write such a concept in Python. The need to fix >> types does constrain things. > > A decorator on the external function declaration, you mean? Something like > this? > > @autoraise(MemoryError) > char* make_new_buffer() except NULL > > Or like this? > > @autoraise(MemoryError, error_value="NULL") > char* make_new_buffer() > > I wouldn't mind. Although there isn't currently a pure Python syntax for > external declarations that would benefit from this ... I think that's too heavyweight for what is essentially magic. > Or did you mean that it should be a user implemented decorator? That would > be nice, but even fused types would make a simple case like this rather > code heavy, I guess. Yeah, that was the vague hope. Even being able to write check(make_new_buffer()[, NULL, MemoryError]) could be another option, if fused types could permit this. Just exploring options, though I'm starting to leaning towards just adding a bit of syntax if this feature is desired. - Robert From stefan_ml at behnel.de Thu Aug 9 14:31:15 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Aug 2012 14:31:15 +0200 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> Message-ID: <5023AD93.2000201@behnel.de> mark florisson, 07.08.2012 11:09: > I thought the 32 bit issue was resolved? You pushed a fix and I fixed > some tests, so it passed for me. I can run it again to check... I don't know. Yaroslav replied to your mail saying that your fixes didn't change anything for the Debian builds. Let's see what he has to say about the second beta. Stefan From lists at onerussian.com Thu Aug 9 17:31:48 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 9 Aug 2012 11:31:48 -0400 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <5023AD93.2000201@behnel.de> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> Message-ID: <20120809153148.GF5866@onerussian.com> On Thu, 09 Aug 2012, Stefan Behnel wrote: > > I thought the 32 bit issue was resolved? You pushed a fix and I fixed > > some tests, so it passed for me. I can run it again to check... > I don't know. Yaroslav replied to your mail saying that your fixes didn't > change anything for the Debian builds. Let's see what he has to say about > the second beta. There might have been a misunderstanding -- I can't recall if I have stated that 32-bit issues were not fixed (but I can be wrong). Probably you meant my reply concerning s390x builds (below)... and I have not uploaded any new revision to Debian since 0.17 beta1. ,--- | Date: Thu, 2 Aug 2012 10:30:32 -0400 | From: Yaroslav Halchenko | To: cython-devel at python.org | Subject: Re: [Cython] [cython-users] Re: Cython 0.17 beta 1 released | | | On Wed, 01 Aug 2012, mark florisson wrote: | > Thanks for the fix. I also pushed a fix for one more test numpy_test | > related to fused types dispatching. That passes all tests for me on 32 | > bit linux. | > > Yaroslav, could you give it a try on the Debian build servers? | | FWIW -- 0.16rc1-550-g8880c78 hasn't added nor fixed any of the failures | on s390x in comparison to 0.16rc1-547-g68811fa | ;-) | `--- -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From stefan_ml at behnel.de Thu Aug 9 17:36:15 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Aug 2012 17:36:15 +0200 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <5023AD93.2000201@behnel.de> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> Message-ID: <5023D8EF.6080904@behnel.de> Stefan Behnel, 09.08.2012 14:31: > mark florisson, 07.08.2012 11:09: >> I thought the 32 bit issue was resolved? You pushed a fix and I fixed >> some tests, so it passed for me. I can run it again to check... > > I don't know. Yaroslav replied to your mail saying that your fixes didn't > change anything for the Debian builds. Let's see what he has to say about > the second beta. He ran them through the build servers. Here are the latest results: https://buildd.debian.org/status/package.php?p=cython&suite=experimental (hint: click on "all" in the Logs column, then click on the build result to see the log output) The i386 tests show 4 errors related to fused types in the NumPy tests: https://buildd.debian.org/status/fetch.php?pkg=cython&arch=i386&ver=0.17~beta1-2&stamp=1343428255 >From a quick look, they might really be problems in the tests rather than the code we generate. It also looks like my signed char fix was not sufficient, as you can see in the memslice tests here, e.g. in the "memslice.__test__.test_memslice_struct_with_arrays" test: https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390&ver=0.17~beta1-2&stamp=1343418358 It now prints """ ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' in 'ArrayStruct.chars' """ Well, "char" and "unsigned char" are supposed to be the same on this platform, but I guess it can't know that. I wonder if "char" and "signed char" are considered identical on other platforms ... To fix the tests, it might be enough to use an explicit "unsigned char" instead of plain "char" for the types. Not sure if that's a good idea or if there actually is a bug hidden below this test failure. At least, it smells like an inconvenience for users to get that error. Stefan From markflorisson88 at gmail.com Thu Aug 9 18:51:01 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 9 Aug 2012 17:51:01 +0100 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <5023D8EF.6080904@behnel.de> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> Message-ID: On 9 August 2012 16:36, Stefan Behnel wrote: > Stefan Behnel, 09.08.2012 14:31: >> mark florisson, 07.08.2012 11:09: >>> I thought the 32 bit issue was resolved? You pushed a fix and I fixed >>> some tests, so it passed for me. I can run it again to check... >> >> I don't know. Yaroslav replied to your mail saying that your fixes didn't >> change anything for the Debian builds. Let's see what he has to say about >> the second beta. > > He ran them through the build servers. Here are the latest results: > > https://buildd.debian.org/status/package.php?p=cython&suite=experimental > > (hint: click on "all" in the Logs column, then click on the build result to > see the log output) > > The i386 tests show 4 errors related to fused types in the NumPy tests: > > https://buildd.debian.org/status/fetch.php?pkg=cython&arch=i386&ver=0.17~beta1-2&stamp=1343428255 > > >From a quick look, they might really be problems in the tests rather than > the code we generate. > > It also looks like my signed char fix was not sufficient, as you can see in > the memslice tests here, e.g. in the > "memslice.__test__.test_memslice_struct_with_arrays" test: > > https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390&ver=0.17~beta1-2&stamp=1343418358 > > It now prints > > """ > ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' > in 'ArrayStruct.chars' > """ > > Well, "char" and "unsigned char" are supposed to be the same on this > platform, but I guess it can't know that. I wonder if "char" and "signed > char" are considered identical on other platforms ... > > To fix the tests, it might be enough to use an explicit "unsigned char" > instead of plain "char" for the types. Not sure if that's a good idea or if > there actually is a bug hidden below this test failure. At least, it smells > like an inconvenience for users to get that error. I think these tests are fine (unlike the fused test I fixed), it's a bug somewhere. I'm not sure where though, because the code that creates the format string uses the type info struct (with the C compile-time unsigned check), and so does the buffer format parser. The code creating the type strings are in the MemoryView.pyx and Buffer.c utility files at the bottom. It does if (type->is_unsigned) *buf = toupper(*buf); So whatever I feed it on my system (char, signed char or unsigned char), it generates the right format string for me. Perhaps we can just ignore these test failures, I don't think many people are using memoryviews with characters. Changing them to signed or unsigned char may fix the problem for the tests, though. Is it bad to release something that doesn't pass the entire test suite on some system? > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Thu Aug 9 19:04:25 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 09 Aug 2012 19:04:25 +0200 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> Message-ID: <5023ED99.8040706@behnel.de> mark florisson, 09.08.2012 18:51: > On 9 August 2012 16:36, Stefan Behnel wrote: >> Stefan Behnel, 09.08.2012 14:31: >>> mark florisson, 07.08.2012 11:09: >>>> I thought the 32 bit issue was resolved? You pushed a fix and I fixed >>>> some tests, so it passed for me. I can run it again to check... >>> >>> I don't know. Yaroslav replied to your mail saying that your fixes didn't >>> change anything for the Debian builds. Let's see what he has to say about >>> the second beta. >> >> He ran them through the build servers. Here are the latest results: >> >> https://buildd.debian.org/status/package.php?p=cython&suite=experimental >> >> (hint: click on "all" in the Logs column, then click on the build result to >> see the log output) >> >> The i386 tests show 4 errors related to fused types in the NumPy tests: >> >> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=i386&ver=0.17~beta1-2&stamp=1343428255 >> >> >From a quick look, they might really be problems in the tests rather than >> the code we generate. >> >> It also looks like my signed char fix was not sufficient, as you can see in >> the memslice tests here, e.g. in the >> "memslice.__test__.test_memslice_struct_with_arrays" test: >> >> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390&ver=0.17~beta1-2&stamp=1343418358 >> >> It now prints >> >> """ >> ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' >> in 'ArrayStruct.chars' >> """ >> >> Well, "char" and "unsigned char" are supposed to be the same on this >> platform, but I guess it can't know that. I wonder if "char" and "signed >> char" are considered identical on other platforms ... >> >> To fix the tests, it might be enough to use an explicit "unsigned char" >> instead of plain "char" for the types. Not sure if that's a good idea or if >> there actually is a bug hidden below this test failure. At least, it smells >> like an inconvenience for users to get that error. > > I think these tests are fine (unlike the fused test I fixed), it's a > bug somewhere. I'm not sure where though, because the code that > creates the format string uses the type info struct (with the C > compile-time unsigned check), and so does the buffer format parser. > The code creating the type strings are in the MemoryView.pyx and > Buffer.c utility files at the bottom. It does > > if (type->is_unsigned) > *buf = toupper(*buf); > > So whatever I feed it on my system (char, signed char or unsigned > char), it generates the right format string for me. I think this needs some more investigation directly on the failing systems. It'll be much easier to figure this out with a debugger. > Perhaps we can just ignore these test failures, I don't think many > people are using memoryviews with characters. Changing them to signed > or unsigned char may fix the problem for the tests, though. If we can't find a way to fix this in time, I'm leaning towards that, too. However, working around a bug in a test is a sure way to forget about the bug and consider everything working. Could you copy the failing tests into a separate test case, disable that in bugs.txt and then change the types in the original test modules to whatever you think would make the tests work? > Is it bad to release something that doesn't pass the entire test suite > on some system? Given that we already made tons of releases without even knowing that they'd fail on some systems, I'd say no. :) So far, we've always taken care of the most common systems and just tried to accommodate for other systems to a certain extent. It really doesn't matter if we fix these things before or after the 0.17 release. Stefan From lists at onerussian.com Thu Aug 9 19:11:02 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 9 Aug 2012 13:11:02 -0400 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <20120809153148.GF5866@onerussian.com> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <20120809153148.GF5866@onerussian.com> Message-ID: <20120809171102.GG5866@onerussian.com> FWIW -- just double checked -- tests pass (of current master) just fine on an i386 Debian system. On Thu, 09 Aug 2012, Yaroslav Halchenko wrote: > On Thu, 09 Aug 2012, Stefan Behnel wrote: > > > I thought the 32 bit issue was resolved? You pushed a fix and I fixed > > > some tests, so it passed for me. I can run it again to check... > > I don't know. Yaroslav replied to your mail saying that your fixes didn't > > change anything for the Debian builds. Let's see what he has to say about > > the second beta. > There might have been a misunderstanding -- I can't recall if I have stated > that 32-bit issues were not fixed (but I can be wrong). Probably you meant my > reply concerning s390x builds (below)... and I have not uploaded any new > revision to Debian since 0.17 beta1. > ,--- > | Date: Thu, 2 Aug 2012 10:30:32 -0400 > | From: Yaroslav Halchenko > | To: cython-devel at python.org > | Subject: Re: [Cython] [cython-users] Re: Cython 0.17 beta 1 released > | On Wed, 01 Aug 2012, mark florisson wrote: > | > Thanks for the fix. I also pushed a fix for one more test numpy_test > | > related to fused types dispatching. That passes all tests for me on 32 > | > bit linux. > | > > Yaroslav, could you give it a try on the Debian build servers? > | FWIW -- 0.16rc1-550-g8880c78 hasn't added nor fixed any of the failures > | on s390x in comparison to 0.16rc1-547-g68811fa > | ;-) > `--- -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From lists at onerussian.com Thu Aug 9 19:14:31 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 9 Aug 2012 13:14:31 -0400 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <5023ED99.8040706@behnel.de> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> Message-ID: <20120809171431.GH5866@onerussian.com> On Thu, 09 Aug 2012, Stefan Behnel wrote: > > Is it bad to release something that doesn't pass the entire test suite > > on some system? > Given that we already made tons of releases without even knowing that > they'd fail on some systems, I'd say no. :) FWIW -- some time ago 0.15.1 build/passed all the tests across all architectures. https://buildd.debian.org/status/package.php?p=cython&suite=sid -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From d.s.seljebotn at astro.uio.no Fri Aug 10 02:36:09 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 10 Aug 2012 02:36:09 +0200 Subject: [Cython] Bitey Message-ID: https://github.com/dabeaz/bitey Dag -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Aug 10 06:23:06 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Aug 2012 06:23:06 +0200 Subject: [Cython] Cython Debian builds - failing cygdb tests In-Reply-To: <20120809230530.GE2464@onerussian.com> References: <50182E8C.1060409@behnel.de> <20120802143031.GC5788@onerussian.com> <5023AE38.9020508@behnel.de> <20120809151512.GA2464@onerussian.com> <5023DF7E.2030806@behnel.de> <20120809170616.GB2464@onerussian.com> <20120809183659.GD5871@onerussian.com> <502410C2.3050704@behnel.de> <20120809230530.GE2464@onerussian.com> Message-ID: <50248CAA.9080605@behnel.de> Hi, Yaroslav Halchenko retried the test run on Sparc and all code related bugs seem to be resolved now (I had to fix a few minor endianess bugs). However, the cygdb tests still fail, but do not lead to a test run failure. Yaroslav Halchenko, 10.08.2012 01:05: > tested current 0.16rc1-591-g5de843a on sparc -- indeed resolved ;) but > then also mentioned that at the beginning there is a separate batch of > tests "inside gdb" which seems to fail... they (some or all) also fail > on i386 where I have started to rerun now after installing gdb. Full > log from sparc: > http://www.onerussian.com/tmp/0.16rc1-591-g5de843a-sparc-tests-output.txt > > Since gdb might or might not be installed (it is not listed in > Build-Depends of the debian package but neither conflicts with having > gdb installed) on the build host -- if those failures could be > just ignored, I better exclude them explicitly (I guess with > --exclude=Debugger) Is that the expected behaviour? Or is there just something missing in the build server setup, maybe something wrong with the gdb Python plugin? It looks like it's doing something, though... Stefan From markflorisson88 at gmail.com Fri Aug 10 12:27:44 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 10 Aug 2012 11:27:44 +0100 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <5023ED99.8040706@behnel.de> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> Message-ID: On 9 August 2012 18:04, Stefan Behnel wrote: > mark florisson, 09.08.2012 18:51: >> On 9 August 2012 16:36, Stefan Behnel wrote: >>> Stefan Behnel, 09.08.2012 14:31: >>>> mark florisson, 07.08.2012 11:09: >>>>> I thought the 32 bit issue was resolved? You pushed a fix and I fixed >>>>> some tests, so it passed for me. I can run it again to check... >>>> >>>> I don't know. Yaroslav replied to your mail saying that your fixes didn't >>>> change anything for the Debian builds. Let's see what he has to say about >>>> the second beta. >>> >>> He ran them through the build servers. Here are the latest results: >>> >>> https://buildd.debian.org/status/package.php?p=cython&suite=experimental >>> >>> (hint: click on "all" in the Logs column, then click on the build result to >>> see the log output) >>> >>> The i386 tests show 4 errors related to fused types in the NumPy tests: >>> >>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=i386&ver=0.17~beta1-2&stamp=1343428255 >>> >>> >From a quick look, they might really be problems in the tests rather than >>> the code we generate. >>> >>> It also looks like my signed char fix was not sufficient, as you can see in >>> the memslice tests here, e.g. in the >>> "memslice.__test__.test_memslice_struct_with_arrays" test: >>> >>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390&ver=0.17~beta1-2&stamp=1343418358 >>> >>> It now prints >>> >>> """ >>> ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' >>> in 'ArrayStruct.chars' >>> """ >>> >>> Well, "char" and "unsigned char" are supposed to be the same on this >>> platform, but I guess it can't know that. I wonder if "char" and "signed >>> char" are considered identical on other platforms ... >>> >>> To fix the tests, it might be enough to use an explicit "unsigned char" >>> instead of plain "char" for the types. Not sure if that's a good idea or if >>> there actually is a bug hidden below this test failure. At least, it smells >>> like an inconvenience for users to get that error. >> >> I think these tests are fine (unlike the fused test I fixed), it's a >> bug somewhere. I'm not sure where though, because the code that >> creates the format string uses the type info struct (with the C >> compile-time unsigned check), and so does the buffer format parser. >> The code creating the type strings are in the MemoryView.pyx and >> Buffer.c utility files at the bottom. It does >> >> if (type->is_unsigned) >> *buf = toupper(*buf); >> >> So whatever I feed it on my system (char, signed char or unsigned >> char), it generates the right format string for me. > > I think this needs some more investigation directly on the failing systems. > It'll be much easier to figure this out with a debugger. > Definitely. Do we have access to those kind of machines, though? >> Perhaps we can just ignore these test failures, I don't think many >> people are using memoryviews with characters. Changing them to signed >> or unsigned char may fix the problem for the tests, though. > > If we can't find a way to fix this in time, I'm leaning towards that, too. > However, working around a bug in a test is a sure way to forget about the > bug and consider everything working. Could you copy the failing tests into > a separate test case, disable that in bugs.txt and then change the types in > the original test modules to whatever you think would make the tests work? > I think the same bug is failing all these cases. I think the thing that's really failing is this: char[:] m = my_char_pointer >> Is it bad to release something that doesn't pass the entire test suite >> on some system? > > Given that we already made tons of releases without even knowing that > they'd fail on some systems, I'd say no. :) > > So far, we've always taken care of the most common systems and just tried > to accommodate for other systems to a certain extent. It really doesn't > matter if we fix these things before or after the 0.17 release. Great :) I'm sorry, but I'm really swamped for the next two weeks (dissertation deadline), so I'm thinking of just leaving these few test failures, they're not really worrisome. > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Fri Aug 10 12:31:15 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 10 Aug 2012 11:31:15 +0100 Subject: [Cython] Cython Debian builds - failing cygdb tests In-Reply-To: <50248CAA.9080605@behnel.de> References: <50182E8C.1060409@behnel.de> <20120802143031.GC5788@onerussian.com> <5023AE38.9020508@behnel.de> <20120809151512.GA2464@onerussian.com> <5023DF7E.2030806@behnel.de> <20120809170616.GB2464@onerussian.com> <20120809183659.GD5871@onerussian.com> <502410C2.3050704@behnel.de> <20120809230530.GE2464@onerussian.com> <50248CAA.9080605@behnel.de> Message-ID: On 10 August 2012 05:23, Stefan Behnel wrote: > Hi, > > Yaroslav Halchenko retried the test run on Sparc and all code related bugs > seem to be resolved now (I had to fix a few minor endianess bugs). > > However, the cygdb tests still fail, but do not lead to a test run failure. > > Yaroslav Halchenko, 10.08.2012 01:05: >> tested current 0.16rc1-591-g5de843a on sparc -- indeed resolved ;) but >> then also mentioned that at the beginning there is a separate batch of >> tests "inside gdb" which seems to fail... they (some or all) also fail >> on i386 where I have started to rerun now after installing gdb. Full >> log from sparc: >> http://www.onerussian.com/tmp/0.16rc1-591-g5de843a-sparc-tests-output.txt >> >> Since gdb might or might not be installed (it is not listed in >> Build-Depends of the debian package but neither conflicts with having >> gdb installed) on the build host -- if those failures could be >> just ignored, I better exclude them explicitly (I guess with >> --exclude=Debugger) > > Is that the expected behaviour? Or is there just something missing in the > build server setup, maybe something wrong with the gdb Python plugin? It > looks like it's doing something, though... IIRC, it parses the gdb version to match against gdb >= 7.2. Something is clearly going wrong, it needs investigation... Unit testing was hard enough on top of dealing with a notoriously unstable Python API, maybe it's using 7.3 and it's backwards incompatible in some subtle way? Or maybe one of the workarounds for gdb bugs were broken by a bug fix? :) No idea really, the tests used to pass with 7.2, no code in the debugger changed really. > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From robertwb at gmail.com Fri Aug 10 18:25:10 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 10 Aug 2012 09:25:10 -0700 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <5023ED99.8040706@behnel.de> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> Message-ID: On Thu, Aug 9, 2012 at 10:04 AM, Stefan Behnel wrote: > mark florisson, 09.08.2012 18:51: >> On 9 August 2012 16:36, Stefan Behnel wrote: >>> Stefan Behnel, 09.08.2012 14:31: >>>> mark florisson, 07.08.2012 11:09: >>>>> I thought the 32 bit issue was resolved? You pushed a fix and I fixed >>>>> some tests, so it passed for me. I can run it again to check... >>>> >>>> I don't know. Yaroslav replied to your mail saying that your fixes didn't >>>> change anything for the Debian builds. Let's see what he has to say about >>>> the second beta. >>> >>> He ran them through the build servers. Here are the latest results: >>> >>> https://buildd.debian.org/status/package.php?p=cython&suite=experimental >>> >>> (hint: click on "all" in the Logs column, then click on the build result to >>> see the log output) >>> >>> The i386 tests show 4 errors related to fused types in the NumPy tests: >>> >>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=i386&ver=0.17~beta1-2&stamp=1343428255 >>> >>> >From a quick look, they might really be problems in the tests rather than >>> the code we generate. >>> >>> It also looks like my signed char fix was not sufficient, as you can see in >>> the memslice tests here, e.g. in the >>> "memslice.__test__.test_memslice_struct_with_arrays" test: >>> >>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390&ver=0.17~beta1-2&stamp=1343418358 >>> >>> It now prints >>> >>> """ >>> ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' >>> in 'ArrayStruct.chars' >>> """ >>> >>> Well, "char" and "unsigned char" are supposed to be the same on this >>> platform, but I guess it can't know that. I wonder if "char" and "signed >>> char" are considered identical on other platforms ... >>> >>> To fix the tests, it might be enough to use an explicit "unsigned char" >>> instead of plain "char" for the types. Not sure if that's a good idea or if >>> there actually is a bug hidden below this test failure. At least, it smells >>> like an inconvenience for users to get that error. >> >> I think these tests are fine (unlike the fused test I fixed), it's a >> bug somewhere. I'm not sure where though, because the code that >> creates the format string uses the type info struct (with the C >> compile-time unsigned check), and so does the buffer format parser. >> The code creating the type strings are in the MemoryView.pyx and >> Buffer.c utility files at the bottom. It does >> >> if (type->is_unsigned) >> *buf = toupper(*buf); >> >> So whatever I feed it on my system (char, signed char or unsigned >> char), it generates the right format string for me. > > I think this needs some more investigation directly on the failing systems. > It'll be much easier to figure this out with a debugger. Strangely enough, I wasn't even able to reproduce this passing "-funsigned-char" so yes, I think it may require access to one of these systems. (Perhaps I'd need to compile Python and/or NumPy with this option as well...) >> Perhaps we can just ignore these test failures, I don't think many >> people are using memoryviews with characters. Changing them to signed >> or unsigned char may fix the problem for the tests, though. > > If we can't find a way to fix this in time, I'm leaning towards that, too. > However, working around a bug in a test is a sure way to forget about the > bug and consider everything working. Could you copy the failing tests into > a separate test case, disable that in bugs.txt and then change the types in > the original test modules to whatever you think would make the tests work? I'll take a stab at this. >> Is it bad to release something that doesn't pass the entire test suite >> on some system? > > Given that we already made tons of releases without even knowing that > they'd fail on some systems, I'd say no. :) > > So far, we've always taken care of the most common systems and just tried > to accommodate for other systems to a certain extent. It really doesn't > matter if we fix these things before or after the 0.17 release. +1 - Robert From robertwb at gmail.com Fri Aug 10 20:54:52 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 10 Aug 2012 11:54:52 -0700 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> Message-ID: On Fri, Aug 10, 2012 at 9:25 AM, Robert Bradshaw wrote: > On Thu, Aug 9, 2012 at 10:04 AM, Stefan Behnel wrote: >> mark florisson, 09.08.2012 18:51: >>> On 9 August 2012 16:36, Stefan Behnel wrote: >>>> Stefan Behnel, 09.08.2012 14:31: >>>>> mark florisson, 07.08.2012 11:09: >>>>>> I thought the 32 bit issue was resolved? You pushed a fix and I fixed >>>>>> some tests, so it passed for me. I can run it again to check... >>>>> >>>>> I don't know. Yaroslav replied to your mail saying that your fixes didn't >>>>> change anything for the Debian builds. Let's see what he has to say about >>>>> the second beta. >>>> >>>> He ran them through the build servers. Here are the latest results: >>>> >>>> https://buildd.debian.org/status/package.php?p=cython&suite=experimental >>>> >>>> (hint: click on "all" in the Logs column, then click on the build result to >>>> see the log output) >>>> >>>> The i386 tests show 4 errors related to fused types in the NumPy tests: >>>> >>>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=i386&ver=0.17~beta1-2&stamp=1343428255 >>>> >>>> >From a quick look, they might really be problems in the tests rather than >>>> the code we generate. >>>> >>>> It also looks like my signed char fix was not sufficient, as you can see in >>>> the memslice tests here, e.g. in the >>>> "memslice.__test__.test_memslice_struct_with_arrays" test: >>>> >>>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390&ver=0.17~beta1-2&stamp=1343418358 >>>> >>>> It now prints >>>> >>>> """ >>>> ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' >>>> in 'ArrayStruct.chars' >>>> """ >>>> >>>> Well, "char" and "unsigned char" are supposed to be the same on this >>>> platform, but I guess it can't know that. I wonder if "char" and "signed >>>> char" are considered identical on other platforms ... >>>> >>>> To fix the tests, it might be enough to use an explicit "unsigned char" >>>> instead of plain "char" for the types. Not sure if that's a good idea or if >>>> there actually is a bug hidden below this test failure. At least, it smells >>>> like an inconvenience for users to get that error. >>> >>> I think these tests are fine (unlike the fused test I fixed), it's a >>> bug somewhere. I'm not sure where though, because the code that >>> creates the format string uses the type info struct (with the C >>> compile-time unsigned check), and so does the buffer format parser. >>> The code creating the type strings are in the MemoryView.pyx and >>> Buffer.c utility files at the bottom. It does >>> >>> if (type->is_unsigned) >>> *buf = toupper(*buf); >>> >>> So whatever I feed it on my system (char, signed char or unsigned >>> char), it generates the right format string for me. >> >> I think this needs some more investigation directly on the failing systems. >> It'll be much easier to figure this out with a debugger. > > Strangely enough, I wasn't even able to reproduce this passing > "-funsigned-char" so yes, I think it may require access to one of > these systems. (Perhaps I'd need to compile Python and/or NumPy with > this option as well...) > >>> Perhaps we can just ignore these test failures, I don't think many >>> people are using memoryviews with characters. Changing them to signed >>> or unsigned char may fix the problem for the tests, though. >> >> If we can't find a way to fix this in time, I'm leaning towards that, too. >> However, working around a bug in a test is a sure way to forget about the >> bug and consider everything working. Could you copy the failing tests into >> a separate test case, disable that in bugs.txt and then change the types in >> the original test modules to whatever you think would make the tests work? > > I'll take a stab at this. OK, the problem boiled down to using 'b' or 'B' for the format string for char rather than 'c'. See, e.g. http://docs.python.org/library/array.html . I've pushed a fix at https://github.com/robertwb/cython/commit/b0539cbc32c200a09b1fbddf2d6943e92aec2f3e , but it'd be good to have another set of eyes with someone more familiar with this code to be sure I didn't miss anything. It also involved a lot of changes to the tests https://github.com/robertwb/cython/commit/6bcc8fd8419e6e4079344788023029736610d5aa so is not exactly backwards compatible, though perhaps more correct. It'd be good to see if this actually fixes the issues. - Robert From lists at onerussian.com Fri Aug 10 20:57:37 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Fri, 10 Aug 2012 14:57:37 -0400 Subject: [Cython] Cython Debian builds - failing cygdb tests In-Reply-To: References: <20120802143031.GC5788@onerussian.com> <5023AE38.9020508@behnel.de> <20120809151512.GA2464@onerussian.com> <5023DF7E.2030806@behnel.de> <20120809170616.GB2464@onerussian.com> <20120809183659.GD5871@onerussian.com> <502410C2.3050704@behnel.de> <20120809230530.GE2464@onerussian.com> <50248CAA.9080605@behnel.de> Message-ID: <20120810185737.GI5866@onerussian.com> On Fri, 10 Aug 2012, mark florisson wrote: > > Is that the expected behaviour? Or is there just something missing in the > > build server setup, maybe something wrong with the gdb Python plugin? It > > looks like it's doing something, though... > IIRC, it parses the gdb version to match against gdb >= 7.2. Something > is clearly going wrong, it needs investigation... FWIW $> gdb --version GNU gdb (GDB) 7.4.1-debian ... seems to be parsed alright by Cython/Debugger/Tests/TestLibCython.py . > Unit testing was > hard enough on top of dealing with a notoriously unstable Python API, > maybe it's using 7.3 and it's backwards incompatible in some subtle > way? could be could be... just in case -- tested with gdb 7.3 -- similar failures. > Or maybe one of the workarounds for gdb bugs were broken by a bug > fix? :) No idea really, the tests used to pass with 7.2, no code in > the debugger changed really. btw -- with runtests.py is there a built-in way to fall into a debugger (pdb) upon failure/exception handling? -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From stefan_ml at behnel.de Fri Aug 10 21:10:16 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Aug 2012 21:10:16 +0200 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> Message-ID: <50255C98.30402@behnel.de> Robert Bradshaw, 10.08.2012 20:54: > OK, the problem boiled down to using 'b' or 'B' for the format string > for char rather than 'c'. See, e.g. > http://docs.python.org/library/array.html . I've pushed a fix at > https://github.com/robertwb/cython/commit/b0539cbc32c200a09b1fbddf2d6943e92aec2f3e > , but it'd be good to have another set of eyes with someone more > familiar with this code to be sure I didn't miss anything. It also > involved a lot of changes to the tests > https://github.com/robertwb/cython/commit/6bcc8fd8419e6e4079344788023029736610d5aa > so is not exactly backwards compatible, though perhaps more correct. That's unfortunate, but since it's a bug, it needs fixing. Stefan From robertwb at gmail.com Fri Aug 10 21:31:27 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 10 Aug 2012 12:31:27 -0700 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <50255C98.30402@behnel.de> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> Message-ID: On Fri, Aug 10, 2012 at 12:10 PM, Stefan Behnel wrote: > Robert Bradshaw, 10.08.2012 20:54: >> OK, the problem boiled down to using 'b' or 'B' for the format string >> for char rather than 'c'. See, e.g. >> http://docs.python.org/library/array.html . I've pushed a fix at >> https://github.com/robertwb/cython/commit/b0539cbc32c200a09b1fbddf2d6943e92aec2f3e >> , but it'd be good to have another set of eyes with someone more >> familiar with this code to be sure I didn't miss anything. It also >> involved a lot of changes to the tests >> https://github.com/robertwb/cython/commit/6bcc8fd8419e6e4079344788023029736610d5aa >> so is not exactly backwards compatible, though perhaps more correct. > > That's unfortunate, but since it's a bug, it needs fixing. Actually, this fix breaks numpy_test, and I'm not sure how to fix it given that numpy has no notion of a "possibly signed char" (unless we disallow using char/char* with numpy altogether, which is much more extreme, or don't enforce that signs match for char, which is likely more invasive). Ugh. - Robert From robertwb at gmail.com Fri Aug 10 22:07:56 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 10 Aug 2012 13:07:56 -0700 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> Message-ID: On Fri, Aug 10, 2012 at 12:31 PM, Robert Bradshaw wrote: > On Fri, Aug 10, 2012 at 12:10 PM, Stefan Behnel wrote: >> Robert Bradshaw, 10.08.2012 20:54: >>> OK, the problem boiled down to using 'b' or 'B' for the format string >>> for char rather than 'c'. See, e.g. >>> http://docs.python.org/library/array.html . I've pushed a fix at >>> https://github.com/robertwb/cython/commit/b0539cbc32c200a09b1fbddf2d6943e92aec2f3e >>> , but it'd be good to have another set of eyes with someone more >>> familiar with this code to be sure I didn't miss anything. It also >>> involved a lot of changes to the tests >>> https://github.com/robertwb/cython/commit/6bcc8fd8419e6e4079344788023029736610d5aa >>> so is not exactly backwards compatible, though perhaps more correct. >> >> That's unfortunate, but since it's a bug, it needs fixing. > > Actually, this fix breaks numpy_test, and I'm not sure how to fix it > given that numpy has no notion of a "possibly signed char" (unless we > disallow using char/char* with numpy altogether, which is much more > extreme, or don't enforce that signs match for char, which is likely > more invasive). Ugh. I suppose the lack of numpy support for (no-signed) char does explain why we were in the state we were in. I've pushed another commit that lets unsigned char == char == signed char for the purposes of buffer comparison (though not transitively, so unsigned char != signed char). - Robert From stefan_ml at behnel.de Fri Aug 10 22:10:38 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Aug 2012 22:10:38 +0200 Subject: [Cython] Anyone for a doc cleanup before the release? In-Reply-To: References: <5020C522.1090900@behnel.de> Message-ID: <50256ABE.7050602@behnel.de> Dimitri Tcaciuc, 09.08.2012 05:40: > On Tue, Aug 7, 2012 at 12:34 AM, Stefan Behnel wrote: >> building the documentation currently yields a number of Sphinx warnings: >> >> https://sage.math.washington.edu:8091/hudson/job/cython-docs/lastSuccessfulBuild/warnings1Result/ >> >> It would be nice if someone found the time to clean some of them up. > > Here's my go at it: > > https://github.com/cython/cython/pull/141 Thanks! I'm reposting your questions here, in case someone has an opinion on them: """ What to do with duplicate citations (Sage, Pyrex). Perhaps make a separate citations page? What to do with sources that are not included anywhere (welcome, userguide/numpy_tutorial, /userguide/pxd_package) public keyword has a dual use; perhaps a section in reference/language basics describing the two would be appropriate? I've also tentatively enabled intersphinx extension to reference standard python keywords, is that OK? """ Stefan From markflorisson88 at gmail.com Fri Aug 10 22:19:38 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 10 Aug 2012 21:19:38 +0100 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> Message-ID: On 10 August 2012 19:54, Robert Bradshaw wrote: > On Fri, Aug 10, 2012 at 9:25 AM, Robert Bradshaw wrote: >> On Thu, Aug 9, 2012 at 10:04 AM, Stefan Behnel wrote: >>> mark florisson, 09.08.2012 18:51: >>>> On 9 August 2012 16:36, Stefan Behnel wrote: >>>>> Stefan Behnel, 09.08.2012 14:31: >>>>>> mark florisson, 07.08.2012 11:09: >>>>>>> I thought the 32 bit issue was resolved? You pushed a fix and I fixed >>>>>>> some tests, so it passed for me. I can run it again to check... >>>>>> >>>>>> I don't know. Yaroslav replied to your mail saying that your fixes didn't >>>>>> change anything for the Debian builds. Let's see what he has to say about >>>>>> the second beta. >>>>> >>>>> He ran them through the build servers. Here are the latest results: >>>>> >>>>> https://buildd.debian.org/status/package.php?p=cython&suite=experimental >>>>> >>>>> (hint: click on "all" in the Logs column, then click on the build result to >>>>> see the log output) >>>>> >>>>> The i386 tests show 4 errors related to fused types in the NumPy tests: >>>>> >>>>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=i386&ver=0.17~beta1-2&stamp=1343428255 >>>>> >>>>> >From a quick look, they might really be problems in the tests rather than >>>>> the code we generate. >>>>> >>>>> It also looks like my signed char fix was not sufficient, as you can see in >>>>> the memslice tests here, e.g. in the >>>>> "memslice.__test__.test_memslice_struct_with_arrays" test: >>>>> >>>>> https://buildd.debian.org/status/fetch.php?pkg=cython&arch=s390&ver=0.17~beta1-2&stamp=1343418358 >>>>> >>>>> It now prints >>>>> >>>>> """ >>>>> ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' >>>>> in 'ArrayStruct.chars' >>>>> """ >>>>> >>>>> Well, "char" and "unsigned char" are supposed to be the same on this >>>>> platform, but I guess it can't know that. I wonder if "char" and "signed >>>>> char" are considered identical on other platforms ... >>>>> >>>>> To fix the tests, it might be enough to use an explicit "unsigned char" >>>>> instead of plain "char" for the types. Not sure if that's a good idea or if >>>>> there actually is a bug hidden below this test failure. At least, it smells >>>>> like an inconvenience for users to get that error. >>>> >>>> I think these tests are fine (unlike the fused test I fixed), it's a >>>> bug somewhere. I'm not sure where though, because the code that >>>> creates the format string uses the type info struct (with the C >>>> compile-time unsigned check), and so does the buffer format parser. >>>> The code creating the type strings are in the MemoryView.pyx and >>>> Buffer.c utility files at the bottom. It does >>>> >>>> if (type->is_unsigned) >>>> *buf = toupper(*buf); >>>> >>>> So whatever I feed it on my system (char, signed char or unsigned >>>> char), it generates the right format string for me. >>> >>> I think this needs some more investigation directly on the failing systems. >>> It'll be much easier to figure this out with a debugger. >> >> Strangely enough, I wasn't even able to reproduce this passing >> "-funsigned-char" so yes, I think it may require access to one of >> these systems. (Perhaps I'd need to compile Python and/or NumPy with >> this option as well...) >> >>>> Perhaps we can just ignore these test failures, I don't think many >>>> people are using memoryviews with characters. Changing them to signed >>>> or unsigned char may fix the problem for the tests, though. >>> >>> If we can't find a way to fix this in time, I'm leaning towards that, too. >>> However, working around a bug in a test is a sure way to forget about the >>> bug and consider everything working. Could you copy the failing tests into >>> a separate test case, disable that in bugs.txt and then change the types in >>> the original test modules to whatever you think would make the tests work? >> >> I'll take a stab at this. > > OK, the problem boiled down to using 'b' or 'B' for the format string > for char rather than 'c'. See, e.g. > http://docs.python.org/library/array.html . That confounds me, I think I need some hand-holding :). So we have a type info struct, from which we create a format string containing 'b' or 'B', depending on signed-ness (at C compile time). We then at buffer format string parsing time use this same type info struct, and we check the size (match) and we check the group of the character in the format string ('b' -> 'I' and 'B' -> 'U') against the expected group from the type info struct (which also depends on the signed-ness). At which point does this go wrong? > I've pushed a fix at > https://github.com/robertwb/cython/commit/b0539cbc32c200a09b1fbddf2d6943e92aec2f3e > , but it'd be good to have another set of eyes with someone more > familiar with this code to be sure I didn't miss anything. It also > involved a lot of changes to the tests > https://github.com/robertwb/cython/commit/6bcc8fd8419e6e4079344788023029736610d5aa > so is not exactly backwards compatible, though perhaps more correct. I think that's ok, people don't typically write these format strings manually. Thanks for the fixes! > It'd be good to see if this actually fixes the issues. > > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Fri Aug 10 22:24:34 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Aug 2012 22:24:34 +0200 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> Message-ID: <50256E02.80301@behnel.de> Robert Bradshaw, 10.08.2012 22:07: > On Fri, Aug 10, 2012 at 12:31 PM, Robert Bradshaw wrote: >> On Fri, Aug 10, 2012 at 12:10 PM, Stefan Behnel wrote: >>> Robert Bradshaw, 10.08.2012 20:54: >>>> OK, the problem boiled down to using 'b' or 'B' for the format string >>>> for char rather than 'c'. See, e.g. >>>> http://docs.python.org/library/array.html . I've pushed a fix at >>>> https://github.com/robertwb/cython/commit/b0539cbc32c200a09b1fbddf2d6943e92aec2f3e >>>> , but it'd be good to have another set of eyes with someone more >>>> familiar with this code to be sure I didn't miss anything. It also >>>> involved a lot of changes to the tests >>>> https://github.com/robertwb/cython/commit/6bcc8fd8419e6e4079344788023029736610d5aa >>>> so is not exactly backwards compatible, though perhaps more correct. >>> >>> That's unfortunate, but since it's a bug, it needs fixing. >> >> Actually, this fix breaks numpy_test, and I'm not sure how to fix it >> given that numpy has no notion of a "possibly signed char" (unless we >> disallow using char/char* with numpy altogether, which is much more >> extreme, or don't enforce that signs match for char, which is likely >> more invasive). Ugh. Many people only run their code on one particular platform (usually some x86 flavour), often just on their own machine. Disallowing it all together sounds overly extreme in that light. > I suppose the lack of numpy support for (no-signed) char does explain > why we were in the state we were in. I've pushed another commit that > lets unsigned char == char == signed char for the purposes of buffer > comparison (though not transitively, so unsigned char != signed char). I find that behaviour acceptable. "char" is underdefined in C and the only really safe value range is 0..127. So we can leave it to users to get it right if they really want to use it. Stefan From brad.froehle at gmail.com Fri Aug 10 22:27:05 2012 From: brad.froehle at gmail.com (Bradley M. Froehle) Date: Fri, 10 Aug 2012 13:27:05 -0700 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> Message-ID: I think again this is an instance of the tests being incorrect, rather than the code. The failing numpy_tests are all cases where we instantiate a buffer of type 'b' (== 'signed char') but then try to use it as if it was a 'char'. The fix, of course is to just replace most instances of 'char' with 'signed char' in numpy_test.pyx: https://github.com/bfroehle/cython/compare/char-buffers In some sense NumPy does have a notion of 'char', namely a 1 character string. The only worry here, of course, is whether this change will painfully break existing code? -Brad > Actually, this fix breaks numpy_test, and I'm not sure how to fix it > given that numpy has no notion of a "possibly signed char" (unless we > disallow using char/char* with numpy altogether, which is much more > extreme, or don't enforce that signs match for char, which is likely > more invasive). Ugh. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Fri Aug 10 22:35:22 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 10 Aug 2012 21:35:22 +0100 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> Message-ID: On 10 August 2012 21:27, Bradley M. Froehle wrote: > I think again this is an instance of the tests being incorrect, rather than > the code. The failing numpy_tests are all cases where we instantiate a > buffer of type 'b' (== 'signed char') but then try to use it as if it was a > 'char'. Yes, those tests are definitely wrong. I'm still wondering about the case where we don't write the format string ourselves, but have it generated, though. > The fix, of course is to just replace most instances of 'char' with 'signed > char' in numpy_test.pyx: > https://github.com/bfroehle/cython/compare/char-buffers > > In some sense NumPy does have a notion of 'char', namely a 1 character > string. > > The only worry here, of course, is whether this change will painfully break > existing code? > > -Brad > > Actually, this fix breaks numpy_test, and I'm not sure how to fix it > given that numpy has no notion of a "possibly signed char" (unless we > disallow using char/char* with numpy altogether, which is much more > extreme, or don't enforce that signs match for char, which is likely > more invasive). Ugh. > > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From d.s.seljebotn at astro.uio.no Fri Aug 10 22:38:56 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 10 Aug 2012 22:38:56 +0200 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> Message-ID: <7fc7654c-764f-48c3-920c-b1ac34a1eac7@email.android.com> I agree. Non-transitive type comparisons seems like very fishy business (it will be *very* surprising to whoever runs across it); I think there's a strong case for just breaking backwards compatability: ERROR: 'char' is illegal as a buffer dtype due to being underspecified in the C standard, please use 'signed char' or 'unsigned char'. (Myself I always use uint8_t or int8_t rather than char...) Dag -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. "Bradley M. Froehle" wrote: I think again this is an instance of the tests being incorrect, rather than the code. The failing numpy_tests are all cases where we instantiate a buffer of type 'b' (== 'signed char') but then try to use it as if it was a 'char'. The fix, of course is to just replace most instances of 'char' with 'signed char' in numpy_test.pyx: https://github.com/bfroehle/cython/compare/char-buffers In some sense NumPy does have a notion of 'char', namely a 1 character string. The only worry here, of course, is whether this change will painfully break existing code? -Brad Actually, this fix breaks numpy_test, and I'm not sure how to fix it given that numpy has no notion of a "possibly signed char" (unless we disallow using char/char* with numpy altogether, which is much more extreme, or don't enforce that signs match for char, which is likely more invasive). Ugh. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Fri Aug 10 23:00:21 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 10 Aug 2012 14:00:21 -0700 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: <7fc7654c-764f-48c3-920c-b1ac34a1eac7@email.android.com> References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> <7fc7654c-764f-48c3-920c-b1ac34a1eac7@email.android.com> Message-ID: We have plenty of non-transitive coercions. E.g. char* <-> object <-> float. While it's technically more correct to use signed or unsigned char, char* is pretty ubiquitous in the C world. There's also the question of the python bytes object and array.array("c") which would be good to support. And this would disallow any nested chars as parts of structs (e.g. from external libraries--you could fake a signness in your declaration but that seems ugly). All in all, particularly in light of interfacing with external code, I think disallowing char is overly strict. On Fri, Aug 10, 2012 at 1:38 PM, Dag Sverre Seljebotn wrote: > I agree. Non-transitive type comparisons seems like very fishy business (it > will be *very* surprising to whoever runs across it); I think there's a > strong case for just breaking backwards compatability: > > ERROR: 'char' is illegal as a buffer dtype due to being underspecified in > the C standard, please use 'signed char' or 'unsigned char'. > > (Myself I always use uint8_t or int8_t rather than char...) > > Dag > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > > "Bradley M. Froehle" wrote: >> >> I think again this is an instance of the tests being incorrect, rather >> than the code. The failing numpy_tests are all cases where we instantiate a >> buffer of type 'b' (== 'signed char') but then try to use it as if it was a >> 'char'. >> >> The fix, of course is to just replace most instances of 'char' with >> 'signed char' in numpy_test.pyx: >> https://github.com/bfroehle/cython/compare/char-buffers >> >> In some sense NumPy does have a notion of 'char', namely a 1 character >> string. >> >> The only worry here, of course, is whether this change will painfully >> break existing code? >> >> -Brad >> >> Actually, this fix breaks numpy_test, and I'm not sure how to fix it >> given that numpy has no notion of a "possibly signed char" (unless we >> disallow using char/char* with numpy altogether, which is much more >> extreme, or don't enforce that signs match for char, which is likely >> more invasive). Ugh. >> >> > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From lists at onerussian.com Sat Aug 11 03:37:18 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Fri, 10 Aug 2012 21:37:18 -0400 Subject: [Cython] Cython Debian builds - failing cygdb tests In-Reply-To: <20120810185737.GI5866@onerussian.com> References: <5023AE38.9020508@behnel.de> <20120809151512.GA2464@onerussian.com> <5023DF7E.2030806@behnel.de> <20120809170616.GB2464@onerussian.com> <20120809183659.GD5871@onerussian.com> <502410C2.3050704@behnel.de> <20120809230530.GE2464@onerussian.com> <50248CAA.9080605@behnel.de> <20120810185737.GI5866@onerussian.com> Message-ID: <20120811013718.GJ5866@onerussian.com> > > IIRC, it parses the gdb version to match against gdb >= 7.2. Something > > is clearly going wrong, it needs investigation... FWIW -- I have built 7.2-1 Debian package (obtained from snapshot.debian.org) under Debian stable squeeze. So I have: ,--- | $> gdb -v | GNU gdb (GDB) 7.2-debian | Copyright (C) 2010 Free Software Foundation, Inc. | License GPLv3+: GNU GPL version 3 or later | This is free software: you are free to change and redistribute it. | There is NO WARRANTY, to the extent permitted by law. Type "show copying" | and "show warranty" for details. | This GDB was configured as "x86_64-linux-gnu". | For bug reporting instructions, please see: | . `--- tests still fail, see http://www.onerussian.com/tmp/0.16rc1-595-g36d8cab-amd64-squeeze-gdb7.2-Debugger-tests-output.txt On Fri, 10 Aug 2012, Yaroslav Halchenko wrote: > > > Is that the expected behaviour? Or is there just something missing in the > > > build server setup, maybe something wrong with the gdb Python plugin? It > > > looks like it's doing something, though... > FWIW > $> gdb --version > GNU gdb (GDB) 7.4.1-debian > ... > seems to be parsed alright by Cython/Debugger/Tests/TestLibCython.py . -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From stefan_ml at behnel.de Sat Aug 11 05:05:00 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Aug 2012 05:05:00 +0200 Subject: [Cython] remaining open issues for 0.17 In-Reply-To: References: <50203DEB.9080108@behnel.de> <5023AD93.2000201@behnel.de> <5023D8EF.6080904@behnel.de> <5023ED99.8040706@behnel.de> <50255C98.30402@behnel.de> <7fc7654c-764f-48c3-920c-b1ac34a1eac7@email.android.com> Message-ID: <5025CBDC.4060308@behnel.de> Robert Bradshaw, 10.08.2012 23:00: > We have plenty of non-transitive coercions. E.g. char* <-> object <-> > float. Python <-> C coercions are not quite the same league as low-level C type comparisons or coercions, though. > While it's technically more correct to use signed or unsigned > char, char* is pretty ubiquitous in the C world. Absolutely - be it due to correct usage, "don't care" or bad design. Requiring explicit signedness can easily drive this into a const-like nightmare with "but I need it" casts all over the place. > There's also the > question of the python bytes object and array.array("c") which would > be good to support. And this would disallow any nested chars as parts > of structs (e.g. from external libraries--you could fake a signness in > your declaration but that seems ugly). And faking declarations could easily lead to even less portable code. > All in all, particularly in light of interfacing with external code, I > think disallowing char is overly strict. +1 Stefan From fperez.net at gmail.com Sat Aug 11 05:40:30 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 10 Aug 2012 20:40:30 -0700 Subject: [Cython] [cython-users] Anyone for a doc cleanup before the release? In-Reply-To: <5020C522.1090900@behnel.de> References: <5020C522.1090900@behnel.de> Message-ID: Question: would you guys be open to including here: http://docs.cython.org/src/quickstart/build.html a note about IPython similar to the Sage one? Something like: """ Similarly, in IPython (as of version 0.13) there is also Cython support that can be used by loading the `cythonmagic` extension. This provides several Cython-related magic functions, the main one being `%%cython` that can be used to prefix blocks of Cython for automatic compilation and importing. This extension can be used at the command line, in the Qt console and in the notebook. """ If you also want an accompanying screenshot, I've made one using the same code and call form as the Sage one for symmetry: http://i.imgur.com/RpUY9.png. If you prefer this in the form of a PR, let me know (and whether you want it with or without figure). Thanks! f From stefan_ml at behnel.de Sat Aug 11 05:58:16 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Aug 2012 05:58:16 +0200 Subject: [Cython] Anyone for a doc cleanup before the release? In-Reply-To: References: <5020C522.1090900@behnel.de> Message-ID: <5025D858.90500@behnel.de> Fernando Perez, 11.08.2012 05:40: > Question: would you guys be open to including here: > > http://docs.cython.org/src/quickstart/build.html > > a note about IPython similar to the Sage one? I think it's either both or none. > Something like: > > """ > Similarly, in IPython (as of version 0.13) there is also Cython > support that can be used by loading the `cythonmagic` extension. This > provides several Cython-related magic functions, the main one being > `%%cython` that can be used to prefix blocks of Cython for automatic > compilation and importing. This extension can be used at the command > line, in the Qt console and in the notebook. > """ > > If you also want an accompanying screenshot, I've made one using the > same code and call form as the Sage one for symmetry: > http://i.imgur.com/RpUY9.png. I was always thinking that that screenshot takes a lot of space on that page compared to the actual text. Looks more like an advertisement and diverts the reader. Maybe we should move both Tools to this page: http://docs.cython.org/src/userguide/source_files_and_compilation.html The quickstart page should then additionally mention cythonize() as an easier alternative to build_ext (although it requires an installed Cython) and can contain a sentence on IPython and the Sage notebook that links to the userguide page. Maybe even more than a sentence because for those who use one of the tools they are the real quickstart. But without the screenshots. > If you prefer this in the form of a PR, let me know (and whether you > want it with or without figure). Yes, once we know what to make of it, please send a pull request. Stefan From fperez.net at gmail.com Sat Aug 11 06:18:56 2012 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 10 Aug 2012 21:18:56 -0700 Subject: [Cython] Anyone for a doc cleanup before the release? In-Reply-To: <5025D858.90500@behnel.de> References: <5020C522.1090900@behnel.de> <5025D858.90500@behnel.de> Message-ID: On Fri, Aug 10, 2012 at 8:58 PM, Stefan Behnel wrote: > Yes, once we know what to make of it, please send a pull request. I'm happy to do so, though I'm not sure I'd be the best one to do real refactoring of your docs if you want to go that way. But just let me know and if I can help out, I'll be happy to do so (it's perfectly OK if you just decide to paste the above paragraph straight in, I'm not looking for commit credit here :) Cheers, f From mongi3 at gmail.com Sat Aug 11 19:27:50 2012 From: mongi3 at gmail.com (Jeff Copeland) Date: Sat, 11 Aug 2012 10:27:50 -0700 Subject: [Cython] Add includes to generated header files In-Reply-To: <501B48E5.2000402@behnel.de> References: <501B48E5.2000402@behnel.de> Message-ID: On Thu, Aug 2, 2012 at 8:43 PM, Stefan Behnel wrote: > That sounds to me like we should make it a configurable option, something > like "capi_reexport_cincludes". Do mean as a command line option, or something else like a compiler directive? From mongi3 at gmail.com Sat Aug 11 20:54:00 2012 From: mongi3 at gmail.com (Jeff Copeland) Date: Sat, 11 Aug 2012 11:54:00 -0700 Subject: [Cython] Add includes to generated header files In-Reply-To: References: <501B48E5.2000402@behnel.de> Message-ID: On Sat, Aug 11, 2012 at 10:27 AM, Jeff Copeland wrote: > On Thu, Aug 2, 2012 at 8:43 PM, Stefan Behnel wrote: >> That sounds to me like we should make it a configurable option, something >> like "capi_reexport_cincludes". > > Do mean as a command line option, or something else like a compiler directive? I went ahead and added a command line option. https://github.com/cython/cython/pull/142 From stefan_ml at behnel.de Sat Aug 11 22:19:53 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 11 Aug 2012 22:19:53 +0200 Subject: [Cython] C++ STL iteration bugs Message-ID: <5026BE69.1020000@behnel.de> Hi, I ran into a couple of problems with the new C++ STL integration, just dumping them here for now. Invalid C code when using a stack allocated C++ vector inside of a generator, also lacking type conversion utility code: https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd Failure to compile iteration over pointer to C++ vector, apparently a type inference bug: https://github.com/cython/cython/commit/edd10c3b33afcac8f471ac6d6664e29f2d985d21 I may not be able to take a serious look at them within the next week but would like to see them fixed for the release, so I'd be happy if someone can manage to work on them. Stefan From robertwb at gmail.com Sun Aug 12 07:03:09 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 11 Aug 2012 22:03:09 -0700 Subject: [Cython] C++ STL iteration bugs In-Reply-To: <5026BE69.1020000@behnel.de> References: <5026BE69.1020000@behnel.de> Message-ID: On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: > Hi, > > I ran into a couple of problems with the new C++ STL integration, just > dumping them here for now. > > Invalid C code when using a stack allocated C++ vector inside of a > generator, also lacking type conversion utility code: > > https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd > > Failure to compile iteration over pointer to C++ vector, apparently a type > inference bug: > > https://github.com/cython/cython/commit/edd10c3b33afcac8f471ac6d6664e29f2d985d21 > > I may not be able to take a serious look at them within the next week but > would like to see them fixed for the release, so I'd be happy if someone > can manage to work on them. I'll be taking a poke at them. - Robert From robertwb at gmail.com Sun Aug 12 07:12:39 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 11 Aug 2012 22:12:39 -0700 Subject: [Cython] C++ STL iteration bugs In-Reply-To: References: <5026BE69.1020000@behnel.de> Message-ID: On Sat, Aug 11, 2012 at 10:03 PM, Robert Bradshaw wrote: > On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: >> Hi, >> >> I ran into a couple of problems with the new C++ STL integration, just >> dumping them here for now. >> >> Invalid C code when using a stack allocated C++ vector inside of a >> generator, also lacking type conversion utility code: >> >> https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd >> >> Failure to compile iteration over pointer to C++ vector, apparently a type >> inference bug: >> >> https://github.com/cython/cython/commit/edd10c3b33afcac8f471ac6d6664e29f2d985d21 Well, the problem is that you're trying to iterate over a vector[int]*, and iteration over pointers already has a meaning. Given v of type vector[int]*, it would be nice to support both iteration over v[a:b] and v (element-wise over the pointed-to vector), but I don't think this is a bug per se. (There is similar ugliness with operator overloading, e.g. given "cdef CppClass* x" should "x + 1" increments the pointer rather than operator+.) - Robert From stefan_ml at behnel.de Sun Aug 12 08:58:56 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 12 Aug 2012 08:58:56 +0200 Subject: [Cython] C++ STL iteration bugs In-Reply-To: References: <5026BE69.1020000@behnel.de> Message-ID: <50275430.5060605@behnel.de> Robert Bradshaw, 12.08.2012 07:12: > On Sat, Aug 11, 2012 at 10:03 PM, Robert Bradshaw wrote: >> On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: >>> I ran into a couple of problems with the new C++ STL integration, just >>> dumping them here for now. >>> >>> Failure to compile iteration over pointer to C++ vector, apparently a type >>> inference bug: >>> >>> https://github.com/cython/cython/commit/edd10c3b33afcac8f471ac6d6664e29f2d985d21 > > Well, the problem is that you're trying to iterate over a > vector[int]*, and iteration over pointers already has a meaning. Ah, sure. > Given v of type vector[int]*, it would be nice to support both iteration > over v[a:b] and v (element-wise over the pointed-to vector), but I > don't think this is a bug per se. Right. However, I don't think iterating over a bare (non-char*) pointer will ever get a meaning, so implementing this by (basically) dereferencing the pointer should be ok. On a somewhat related note, it would be nice to support for x in iter(c_ptr, end_value): ... That would mostly be a generalisation of the special "ends with c'\0'" special case of char* iteration, i.e. that would become equivalent to iter(char_ptr, c'\0') > (There is similar ugliness with > operator overloading, e.g. given "cdef CppClass* x" should "x + 1" > increments the pointer rather than operator+.) Hmm, right. That's less ugly though, because the pointer increment actually makes sense. Stefan From robertwb at gmail.com Sun Aug 12 09:00:29 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Sun, 12 Aug 2012 00:00:29 -0700 Subject: [Cython] C++ STL iteration bugs In-Reply-To: <5026BE69.1020000@behnel.de> References: <5026BE69.1020000@behnel.de> Message-ID: On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: > Hi, > > I ran into a couple of problems with the new C++ STL integration, just > dumping them here for now. > > Invalid C code when using a stack allocated C++ vector inside of a > generator, also lacking type conversion utility code: > > https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd The problem here is that utility code attached at this stage of the pipeline gets ignored. In particular, we get to https://github.com/cython/cython/blob/a7c689c10eea66f9fe384d7bc324a7aa50975f9d/Cython/Compiler/UtilityCode.py#L130 which I am at a loss to explain. - Robert From stefan_ml at behnel.de Sun Aug 12 10:20:03 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 12 Aug 2012 10:20:03 +0200 Subject: [Cython] C++ STL iteration bugs In-Reply-To: <50275430.5060605@behnel.de> References: <5026BE69.1020000@behnel.de> <50275430.5060605@behnel.de> Message-ID: <50276733.4030802@behnel.de> Stefan Behnel, 12.08.2012 08:58: > On a somewhat related note, it would be nice to support > > for x in iter(c_ptr, end_value): > ... > > That would mostly be a generalisation of the special "ends with c'\0'" > special case of char* iteration, i.e. that would become equivalent to > > iter(char_ptr, c'\0') http://trac.cython.org/cython_trac/ticket/786 Stefan From markflorisson88 at gmail.com Sun Aug 12 12:26:20 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 12 Aug 2012 11:26:20 +0100 Subject: [Cython] C++ STL iteration bugs In-Reply-To: References: <5026BE69.1020000@behnel.de> Message-ID: On 12 August 2012 08:00, Robert Bradshaw wrote: > On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: >> Hi, >> >> I ran into a couple of problems with the new C++ STL integration, just >> dumping them here for now. >> >> Invalid C code when using a stack allocated C++ vector inside of a >> generator, also lacking type conversion utility code: >> >> https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd > > The problem here is that utility code attached at this stage of the > pipeline gets ignored. In particular, we get to > https://github.com/cython/cython/blob/a7c689c10eea66f9fe384d7bc324a7aa50975f9d/Cython/Compiler/UtilityCode.py#L130 > which I am at a loss to explain. Heh, that should have had a comment :) Cython utility codes are inserted at the AST level in ModuleNode (the merge_in method). This is done since there is no actual C code at that point. I think it could generate it and capture it as well, but it's probably more complicated, and this works well with the name mangling schemes and such. > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From robertwb at gmail.com Mon Aug 13 17:42:47 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 13 Aug 2012 08:42:47 -0700 Subject: [Cython] C++ STL iteration bugs In-Reply-To: References: <5026BE69.1020000@behnel.de> Message-ID: On Sun, Aug 12, 2012 at 3:26 AM, mark florisson wrote: > On 12 August 2012 08:00, Robert Bradshaw wrote: >> On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: >>> Hi, >>> >>> I ran into a couple of problems with the new C++ STL integration, just >>> dumping them here for now. >>> >>> Invalid C code when using a stack allocated C++ vector inside of a >>> generator, also lacking type conversion utility code: >>> >>> https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd >> >> The problem here is that utility code attached at this stage of the >> pipeline gets ignored. In particular, we get to >> https://github.com/cython/cython/blob/a7c689c10eea66f9fe384d7bc324a7aa50975f9d/Cython/Compiler/UtilityCode.py#L130 >> which I am at a loss to explain. > > Heh, that should have had a comment :) Cython utility codes are > inserted at the AST level in ModuleNode (the merge_in method). This is > done since there is no actual C code at that point. I think it could > generate it and capture it as well, but it's probably more > complicated, and this works well with the name mangling schemes and > such. I figured there was a good explanation :). For the moment, we could probably work around it by calling the relevant functions earlier in the pipeline, but it would be nice to get rid of this limitation. - Robert From markflorisson88 at gmail.com Mon Aug 13 18:25:40 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Mon, 13 Aug 2012 17:25:40 +0100 Subject: [Cython] C++ STL iteration bugs In-Reply-To: References: <5026BE69.1020000@behnel.de> Message-ID: On 13 August 2012 16:42, Robert Bradshaw wrote: > On Sun, Aug 12, 2012 at 3:26 AM, mark florisson > wrote: >> On 12 August 2012 08:00, Robert Bradshaw wrote: >>> On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: >>>> Hi, >>>> >>>> I ran into a couple of problems with the new C++ STL integration, just >>>> dumping them here for now. >>>> >>>> Invalid C code when using a stack allocated C++ vector inside of a >>>> generator, also lacking type conversion utility code: >>>> >>>> https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd >>> >>> The problem here is that utility code attached at this stage of the >>> pipeline gets ignored. In particular, we get to >>> https://github.com/cython/cython/blob/a7c689c10eea66f9fe384d7bc324a7aa50975f9d/Cython/Compiler/UtilityCode.py#L130 >>> which I am at a loss to explain. >> >> Heh, that should have had a comment :) Cython utility codes are >> inserted at the AST level in ModuleNode (the merge_in method). This is >> done since there is no actual C code at that point. I think it could >> generate it and capture it as well, but it's probably more >> complicated, and this works well with the name mangling schemes and >> such. > > I figured there was a good explanation :). For the moment, we could > probably work around it by calling the relevant functions earlier in > the pipeline, but it would be nice to get rid of this limitation. Yeah, it may work to just pass in the code object or the right insertion point to the code generation function of the AST. I inherited the code and never really touched that part, no idea if that will break things. > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From matthew.brett at gmail.com Wed Aug 15 20:38:23 2012 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 15 Aug 2012 11:38:23 -0700 Subject: [Cython] In-place division in memoryview, ndarray causes compiler error Message-ID: Hi, For this file: def div1(int[:] A): A[0] /= 1 or this one: def div2(object[int, ndim=1] A): A[0] /= 1 I get: File "/Users/mb312/usr/local/lib/python2.7/site-packages/Cython/Compiler/Nodes.py", line 354, in generate_execution_code stat.generate_execution_code(code) File "/Users/mb312/usr/local/lib/python2.7/site-packages/Cython/Compiler/Nodes.py", line 4536, in generate_execution_code if c_op in ('/', '%') and self.lhs.type.is_int and not code.directives['cdivision']: AttributeError: 'CCodeWriter' object has no attribute 'directives This is only for in-place division. I'm using current trunk: Cython version 0.17b2 Cheers, Matthew From brett.calcott at gmail.com Mon Aug 20 08:23:22 2012 From: brett.calcott at gmail.com (Brett Calcott) Date: Mon, 20 Aug 2012 16:23:22 +1000 Subject: [Cython] Templated attributes in extensions classes in 0.17 Beta 2 Message-ID: I'm not sure if this is a bug in 0.17 beta or not. But something is not working. Default-constructed attributes in an extension definition appear to work (though I can't see where they are documented on the site?) So this works: ----test.h -------- #include struct Attributes { std::vector v; }; -----test.pyx------ from libcpp.vector cimport vector cdef extern from "test.h": cdef cppclass Attributes: vector[int] v cdef class Test: cdef Attributes a def __cinit__(self, list x): self.a.v = x However, if I try to put the templated attribute in directly: ----test2.pyx------ from libcpp.vector cimport vector cdef class Test: cdef vector[int] v def __cinit__(self, list x): self.v = x cython compiles it fine, but I get an error in the cpp compilation: test2.cpp:722:23: error: expected a type new((void*)&(p->v)) std::vector(); So it looks like it's doing the funky in place construction, but not getting the template parameters in there? Should this work? Cheers, Brett From stefan_ml at behnel.de Mon Aug 20 08:41:04 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Aug 2012 08:41:04 +0200 Subject: [Cython] Templated attributes in extensions classes in 0.17 Beta 2 In-Reply-To: References: Message-ID: <5031DC00.1090702@behnel.de> Brett Calcott, 20.08.2012 08:23: > I'm not sure if this is a bug in 0.17 beta or not. But something is not working. > > Default-constructed attributes in an extension definition appear to > work (though I can't see where they are documented on the site?) > > So this works: > > ----test.h -------- > #include > > struct Attributes > { > std::vector v; > }; > > -----test.pyx------ > from libcpp.vector cimport vector > > cdef extern from "test.h": > cdef cppclass Attributes: > vector[int] v > > cdef class Test: > cdef Attributes a > def __cinit__(self, list x): > self.a.v = x > > > However, if I try to put the templated attribute in directly: > > ----test2.pyx------ > from libcpp.vector cimport vector > > cdef class Test: > cdef vector[int] v > def __cinit__(self, list x): > self.v = x > > cython compiles it fine, but I get an error in the cpp compilation: > test2.cpp:722:23: error: expected a type > new((void*)&(p->v)) std::vector(); > > So it looks like it's doing the funky in place construction, but not > getting the template parameters in there? > > Should this work? Yes, although (IIRC) not in beta 2. There was a recent fix for this, but it broke other stuff. I'm waiting for a fix for those before I release yet another beta (which will then hopefully be a release candidate). In any case, thanks for the report! Stefan From stefan_ml at behnel.de Mon Aug 20 09:39:42 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Aug 2012 09:39:42 +0200 Subject: [Cython] In-place division in memoryview, ndarray causes compiler error In-Reply-To: References: Message-ID: <5031E9BE.7000603@behnel.de> Matthew Brett, 15.08.2012 20:38: > For this file: > > > def div1(int[:] A): > A[0] /= 1 > > > or this one: > > > def div2(object[int, ndim=1] A): > A[0] /= 1 > > > I get: > > File "/Users/mb312/usr/local/lib/python2.7/site-packages/Cython/Compiler/Nodes.py", > line 354, in generate_execution_code > stat.generate_execution_code(code) > File "/Users/mb312/usr/local/lib/python2.7/site-packages/Cython/Compiler/Nodes.py", > line 4536, in generate_execution_code > if c_op in ('/', '%') and self.lhs.type.is_int and not > code.directives['cdivision']: > AttributeError: 'CCodeWriter' object has no attribute 'directives > > This is only for in-place division. I'm using current trunk: > > Cython version 0.17b2 Thanks for the report, here is a fix: https://github.com/cython/cython/commit/7551bd228685329e4309a56939b11d3c98791ede Stefan From stefan_ml at behnel.de Mon Aug 20 15:39:47 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Aug 2012 15:39:47 +0200 Subject: [Cython] new arrayarray.h header file not C99 compatible Message-ID: <50323E23.5080609@behnel.de> Hi, I'm getting test build failures with the new arrayarray.h header file when I enable "--std=c89" or "c99" in the CFLAGS: """ memoryview_inplace_division.c:877: warning: declaration does not declare anything memoryview_inplace_division.c:893: warning: declaration does not declare anything memoryview_inplace_division.c: In function ?newarrayobject?: memoryview_inplace_division.c:929: error: ?arrayobject? has no member named ?ob_item? ... """ The lines it warns about (and which trigger the subsequent errors) are the following union declarations: """ typedef struct arrayobject { PyObject_HEAD union { Py_ssize_t ob_size, length; }; union { char *ob_item; float *_f; double *_d; int *_i; unsigned *_I; unsigned char *_B; signed char *_b; char *_c; unsigned long *_L; long *_l; short *_h; unsigned short *_H; Py_UNICODE *_u; void *_v; }; ... """ Apparently, anonymous unions only became part of the C standard in C11: http://stackoverflow.com/questions/3228104/anonymous-union-within-struct-not-in-c99 That's unfortunate, but I think we'd best give the unions a name (and let users change all code that uses them ...). BTW, is there any reason we have a "length" field at all? Shouldn't we be using Py_SIZE() ? Stefan From stefan_ml at behnel.de Mon Aug 20 15:47:04 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Aug 2012 15:47:04 +0200 Subject: [Cython] new arrayarray.h header file not C99 compatible In-Reply-To: <50323E23.5080609@behnel.de> References: <50323E23.5080609@behnel.de> Message-ID: <50323FD8.2090305@behnel.de> Stefan Behnel, 20.08.2012 15:39: > I'm getting test build failures with the new arrayarray.h header file when > I enable "--std=c89" or "c99" in the CFLAGS: > > """ > memoryview_inplace_division.c:877: warning: declaration does not declare > anything > memoryview_inplace_division.c:893: warning: declaration does not declare > anything > memoryview_inplace_division.c: In function ?newarrayobject?: > memoryview_inplace_division.c:929: error: ?arrayobject? has no member named > ?ob_item? > ... > """ > > The lines it warns about (and which trigger the subsequent errors) are the > following union declarations: > > """ > typedef struct arrayobject { > PyObject_HEAD > union { > Py_ssize_t ob_size, length; > }; > union { > char *ob_item; > float *_f; > double *_d; > int *_i; > unsigned *_I; > unsigned char *_B; > signed char *_b; > char *_c; > unsigned long *_L; > long *_l; > short *_h; > unsigned short *_H; > Py_UNICODE *_u; > void *_v; > }; > ... > """ > > Apparently, anonymous unions only became part of the C standard in C11: > > http://stackoverflow.com/questions/3228104/anonymous-union-within-struct-not-in-c99 > > That's unfortunate, but I think we'd best give the unions a name (and let > users change all code that uses them ...). Hmm, although - our test runner has this: """ def update_pyarray_extension(ext): # See http://gcc.gnu.org/onlinedocs/gcc/Unnamed-Fields.html#Unnamed-Fields ext.extra_compile_args.append('-fms-extensions') """ Is that really the way it should be handled? > BTW, is there any reason we have a "length" field at all? Shouldn't we be > using Py_SIZE() ? Stefan From robertwb at gmail.com Mon Aug 20 17:54:20 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 20 Aug 2012 08:54:20 -0700 Subject: [Cython] new arrayarray.h header file not C99 compatible In-Reply-To: <50323E23.5080609@behnel.de> References: <50323E23.5080609@behnel.de> Message-ID: On Mon, Aug 20, 2012 at 6:39 AM, Stefan Behnel wrote: > Hi, > > I'm getting test build failures with the new arrayarray.h header file when > I enable "--std=c89" or "c99" in the CFLAGS: > > """ > memoryview_inplace_division.c:877: warning: declaration does not declare > anything > memoryview_inplace_division.c:893: warning: declaration does not declare > anything > memoryview_inplace_division.c: In function ?newarrayobject?: > memoryview_inplace_division.c:929: error: ?arrayobject? has no member named > ?ob_item? > ... > """ > > The lines it warns about (and which trigger the subsequent errors) are the > following union declarations: > > """ > typedef struct arrayobject { > PyObject_HEAD > union { > Py_ssize_t ob_size, length; > }; > union { > char *ob_item; > float *_f; > double *_d; > int *_i; > unsigned *_I; > unsigned char *_B; > signed char *_b; > char *_c; > unsigned long *_L; > long *_l; > short *_h; > unsigned short *_H; > Py_UNICODE *_u; > void *_v; > }; > ... > """ > > Apparently, anonymous unions only became part of the C standard in C11: > > http://stackoverflow.com/questions/3228104/anonymous-union-within-struct-not-in-c99 > > That's unfortunate, but I think we'd best give the unions a name (and let > users change all code that uses them ...). I suppose we could do this. Actually, my_array.data.* isn't that bad. (Would it be to verbose to do "my_array.data.as_doubles" while we're on the topic of changing names?) > BTW, is there any reason we have a "length" field at all? Shouldn't we be > using Py_SIZE() ? I just kept the code as it was (which is why I added the extra compile args), but wouldn't be opposed to this change either. Supporting itemsize directly might be more useful. - Robert From stefan_ml at behnel.de Mon Aug 20 18:10:58 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Aug 2012 18:10:58 +0200 Subject: [Cython] new arrayarray.h header file not C99 compatible In-Reply-To: References: <50323E23.5080609@behnel.de> Message-ID: <50326192.40003@behnel.de> Robert Bradshaw, 20.08.2012 17:54: > On Mon, Aug 20, 2012 at 6:39 AM, Stefan Behnel wrote: >> I'm getting test build failures with the new arrayarray.h header file when >> I enable "--std=c89" or "c99" in the CFLAGS: >> >> """ >> memoryview_inplace_division.c:877: warning: declaration does not declare >> anything >> memoryview_inplace_division.c:893: warning: declaration does not declare >> anything >> memoryview_inplace_division.c: In function ?newarrayobject?: >> memoryview_inplace_division.c:929: error: ?arrayobject? has no member named >> ?ob_item? >> ... >> """ >> >> The lines it warns about (and which trigger the subsequent errors) are the >> following union declarations: >> >> """ >> typedef struct arrayobject { >> PyObject_HEAD >> union { >> Py_ssize_t ob_size, length; >> }; >> union { >> char *ob_item; >> float *_f; >> double *_d; >> int *_i; >> unsigned *_I; >> unsigned char *_B; >> signed char *_b; >> char *_c; >> unsigned long *_L; >> long *_l; >> short *_h; >> unsigned short *_H; >> Py_UNICODE *_u; >> void *_v; >> }; >> ... >> """ >> >> Apparently, anonymous unions only became part of the C standard in C11: >> >> http://stackoverflow.com/questions/3228104/anonymous-union-within-struct-not-in-c99 >> >> That's unfortunate, but I think we'd best give the unions a name (and let >> users change all code that uses them ...). > > I suppose we could do this. Actually, my_array.data.* isn't that bad. > (Would it be to verbose to do "my_array.data.as_doubles" while we're > on the topic of changing names?) +1 - people who care about shortcuts will use memory views anyway. I'll change it. >> BTW, is there any reason we have a "length" field at all? Shouldn't we be >> using Py_SIZE() ? > > I just kept the code as it was (which is why I added the extra compile > args), but wouldn't be opposed to this change either. Supporting > itemsize directly might be more useful. Hmm - how would you make "self->ob_descr->itemsize" available more directly? Stefan From robertwb at gmail.com Mon Aug 20 19:44:48 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 20 Aug 2012 10:44:48 -0700 Subject: [Cython] new arrayarray.h header file not C99 compatible In-Reply-To: <50326192.40003@behnel.de> References: <50323E23.5080609@behnel.de> <50326192.40003@behnel.de> Message-ID: On Mon, Aug 20, 2012 at 9:10 AM, Stefan Behnel wrote: > Robert Bradshaw, 20.08.2012 17:54: >> On Mon, Aug 20, 2012 at 6:39 AM, Stefan Behnel wrote: >>> I'm getting test build failures with the new arrayarray.h header file when >>> I enable "--std=c89" or "c99" in the CFLAGS: >>> >>> """ >>> memoryview_inplace_division.c:877: warning: declaration does not declare >>> anything >>> memoryview_inplace_division.c:893: warning: declaration does not declare >>> anything >>> memoryview_inplace_division.c: In function ?newarrayobject?: >>> memoryview_inplace_division.c:929: error: ?arrayobject? has no member named >>> ?ob_item? >>> ... >>> """ >>> >>> The lines it warns about (and which trigger the subsequent errors) are the >>> following union declarations: >>> >>> """ >>> typedef struct arrayobject { >>> PyObject_HEAD >>> union { >>> Py_ssize_t ob_size, length; >>> }; >>> union { >>> char *ob_item; >>> float *_f; >>> double *_d; >>> int *_i; >>> unsigned *_I; >>> unsigned char *_B; >>> signed char *_b; >>> char *_c; >>> unsigned long *_L; >>> long *_l; >>> short *_h; >>> unsigned short *_H; >>> Py_UNICODE *_u; >>> void *_v; >>> }; >>> ... >>> """ >>> >>> Apparently, anonymous unions only became part of the C standard in C11: >>> >>> http://stackoverflow.com/questions/3228104/anonymous-union-within-struct-not-in-c99 >>> >>> That's unfortunate, but I think we'd best give the unions a name (and let >>> users change all code that uses them ...). >> >> I suppose we could do this. Actually, my_array.data.* isn't that bad. >> (Would it be to verbose to do "my_array.data.as_doubles" while we're >> on the topic of changing names?) > > +1 - people who care about shortcuts will use memory views anyway. > > I'll change it. > > >>> BTW, is there any reason we have a "length" field at all? Shouldn't we be >>> using Py_SIZE() ? >> >> I just kept the code as it was (which is why I added the extra compile >> args), but wouldn't be opposed to this change either. Supporting >> itemsize directly might be more useful. > > Hmm - how would you make "self->ob_descr->itemsize" available more directly? I assume it's a function of Py_SIZE and sizeof(PyObject_HEAD), but that's all the thinking I did about it. - Robert From stefan_ml at behnel.de Mon Aug 20 20:55:13 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Aug 2012 20:55:13 +0200 Subject: [Cython] C++ STL iteration bugs In-Reply-To: References: <5026BE69.1020000@behnel.de> Message-ID: <50328811.2050202@behnel.de> Robert Bradshaw, 12.08.2012 09:00: > On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: >> I ran into a couple of problems with the new C++ STL integration, just >> dumping them here for now. >> >> Invalid C code when using a stack allocated C++ vector inside of a >> generator, also lacking type conversion utility code: >> >> https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd > > The problem here is that utility code attached at this stage of the > pipeline gets ignored. In particular, we get to > https://github.com/cython/cython/blob/a7c689c10eea66f9fe384d7bc324a7aa50975f9d/Cython/Compiler/UtilityCode.py#L130 > which I am at a loss to explain. I debugged through this a bit and it seems to me that the main problem is that create_from_py_utility_code() is called too late. Currently, it's the code generation phase that calls it (Node.py, around line 2945). That's ok as long as it only generates C code, but it's way too late for Cython utility code at that point. IIRC, the reason why we call it so late is that type conversions can change during several pipeline phases, even way after type analysis, and we want to make sure that we only serialise utility code that is really needed. Sadly, that applies to Cython utility code equally well. Not sure what the right fix it here. I mean, we could call it earlier, even during type analysis, but that might let us generate code that is never needed, thus triggering a couple of C compiler warnings. Maybe that's acceptable at this point (better too much code than incomplete code). We could also try to special case Cython utility code at some place, call create_from_py_utility_code() early and prevent all other types of utility code objects from being injecting into the env at that time, before the code generation phase. Something like that. Stefan From stefan_ml at behnel.de Mon Aug 20 21:27:34 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Aug 2012 21:27:34 +0200 Subject: [Cython] C++ STL iteration bugs In-Reply-To: <50328811.2050202@behnel.de> References: <5026BE69.1020000@behnel.de> <50328811.2050202@behnel.de> Message-ID: <50328FA6.4060604@behnel.de> Stefan Behnel, 20.08.2012 20:55: > Robert Bradshaw, 12.08.2012 09:00: >> On Sat, Aug 11, 2012 at 1:19 PM, Stefan Behnel wrote: >>> I ran into a couple of problems with the new C++ STL integration, just >>> dumping them here for now. >>> >>> Invalid C code when using a stack allocated C++ vector inside of a >>> generator, also lacking type conversion utility code: >>> >>> https://github.com/cython/cython/commit/d0a8e24720c3ed4e723b5a1b204bf375780f51bd >> >> The problem here is that utility code attached at this stage of the >> pipeline gets ignored. In particular, we get to >> https://github.com/cython/cython/blob/a7c689c10eea66f9fe384d7bc324a7aa50975f9d/Cython/Compiler/UtilityCode.py#L130 >> which I am at a loss to explain. > > I debugged through this a bit and it seems to me that the main problem is > that create_from_py_utility_code() is called too late. Currently, it's the > code generation phase that calls it (Node.py, around line 2945). That's ok > as long as it only generates C code, but it's way too late for Cython > utility code at that point. > > IIRC, the reason why we call it so late is that type conversions can change > during several pipeline phases, even way after type analysis, and we want > to make sure that we only serialise utility code that is really needed. > Sadly, that applies to Cython utility code equally well. > > Not sure what the right fix it here. I mean, we could call it earlier, even > during type analysis, but that might let us generate code that is never > needed, thus triggering a couple of C compiler warnings. Maybe that's > acceptable at this point (better too much code than incomplete code). It seems to work if I call it in DefNodeWrapper.analyse_declarations(). Given that we do not do any fancy stuff on argument types anyway, e.g. no type inference, I don't think this can really hurt (but a test run is still pending...). Stefan From stefan_ml at behnel.de Tue Aug 21 08:18:50 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 21 Aug 2012 08:18:50 +0200 Subject: [Cython] two remaining test failures before the release Message-ID: <5033284A.4050700@behnel.de> Hi, we are now down to two test failures in the NumPy memoryview test: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/647/testReport/ I'm going to make a "final" :) pre-release when they are fixed, so please have a look. Stefan From robertwb at gmail.com Tue Aug 21 12:54:27 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 21 Aug 2012 03:54:27 -0700 Subject: [Cython] two remaining test failures before the release In-Reply-To: <5033284A.4050700@behnel.de> References: <5033284A.4050700@behnel.de> Message-ID: I fixed one of them by making char signed. The root issue seems to be that when trying to convert an object to numpy, it looks at the buffer format, and rejects plain char. The fallback code (using struct et al) is unable to cope with the nested/repeated formats. The other could be fixed similarly, if that's the direction we want to go. - Robert On Mon, Aug 20, 2012 at 11:18 PM, Stefan Behnel wrote: > Hi, > > we are now down to two test failures in the NumPy memoryview test: > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/647/testReport/ > > I'm going to make a "final" :) pre-release when they are fixed, so please > have a look. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Tue Aug 21 13:43:17 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 21 Aug 2012 12:43:17 +0100 Subject: [Cython] two remaining test failures before the release In-Reply-To: References: <5033284A.4050700@behnel.de> Message-ID: Heh, so it seems if numpy doesn't understand the format string it falls back to copying the array through the sequence protocol (__getitem__ + len)? Since Cython can't convert multi-dimension C arrays to objects, the memoryview getitem will use the struct module (which really handles a subset of the things Cython handles). It cannot handle any type of struct. On 21 August 2012 11:54, Robert Bradshaw wrote: > I fixed one of them by making char signed. The root issue seems to be > that when trying to convert an object to numpy, it looks at the buffer > format, and rejects plain char. The fallback code (using struct et al) > is unable to cope with the nested/repeated formats. The other could be > fixed similarly, if that's the direction we want to go. > > - Robert > > > On Mon, Aug 20, 2012 at 11:18 PM, Stefan Behnel wrote: >> Hi, >> >> we are now down to two test failures in the NumPy memoryview test: >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/647/testReport/ >> >> I'm going to make a "final" :) pre-release when they are fixed, so please >> have a look. >> >> Stefan >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Tue Aug 21 17:09:42 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 21 Aug 2012 17:09:42 +0200 Subject: [Cython] two remaining test failures before the release In-Reply-To: References: <5033284A.4050700@behnel.de> Message-ID: <5033A4B6.4010001@behnel.de> mark florisson, 21.08.2012 13:43: > On 21 August 2012 11:54, Robert Bradshaw wrote: >> On Mon, Aug 20, 2012 at 11:18 PM, Stefan Behnel wrote: >>> we are now down to two test failures in the NumPy memoryview test: >>> >>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/647/testReport/ >>> >> I fixed one of them by making char signed. The root issue seems to be >> that when trying to convert an object to numpy, it looks at the buffer >> format, and rejects plain char. The fallback code (using struct et al) >> is unable to cope with the nested/repeated formats. The other could be >> fixed similarly, if that's the direction we want to go. > > Heh, so it seems if numpy doesn't understand the format string it > falls back to copying the array through the sequence protocol > (__getitem__ + len)? Since Cython can't convert multi-dimension C > arrays to objects, the memoryview getitem will use the struct module > (which really handles a subset of the things Cython handles). It > cannot handle any type of struct. Ok - anyone objecting to just fixing the tests for the release? Users will tell us early enough if their code stopped working. Stefan From markflorisson88 at gmail.com Tue Aug 21 17:19:48 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 21 Aug 2012 16:19:48 +0100 Subject: [Cython] two remaining test failures before the release In-Reply-To: <5033A4B6.4010001@behnel.de> References: <5033284A.4050700@behnel.de> <5033A4B6.4010001@behnel.de> Message-ID: On 21 August 2012 16:09, Stefan Behnel wrote: > mark florisson, 21.08.2012 13:43: >> On 21 August 2012 11:54, Robert Bradshaw wrote: >>> On Mon, Aug 20, 2012 at 11:18 PM, Stefan Behnel wrote: >>>> we are now down to two test failures in the NumPy memoryview test: >>>> >>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/647/testReport/ >>>> >>> I fixed one of them by making char signed. The root issue seems to be >>> that when trying to convert an object to numpy, it looks at the buffer >>> format, and rejects plain char. The fallback code (using struct et al) >>> is unable to cope with the nested/repeated formats. The other could be >>> fixed similarly, if that's the direction we want to go. >> >> Heh, so it seems if numpy doesn't understand the format string it >> falls back to copying the array through the sequence protocol >> (__getitem__ + len)? Since Cython can't convert multi-dimension C >> arrays to objects, the memoryview getitem will use the struct module >> (which really handles a subset of the things Cython handles). It >> cannot handle any type of struct. > > Ok - anyone objecting to just fixing the tests for the release? Users will > tell us early enough if their code stopped working. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel I think that's the right approach, considering this is not a bug on our side. From vitja.makarov at gmail.com Wed Aug 22 06:51:44 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 22 Aug 2012 08:51:44 +0400 Subject: [Cython] Jenkins problem Message-ID: Hi! I've a problem running jenknins job https://sage.math.washington.edu:8091/hudson/view/dev-vitek/job/cython-vitek-tests/137/BACKEND=c,PYVERSION=py27/console It says "No space left on device". Have something change there? -- vitja. From stefan_ml at behnel.de Wed Aug 22 07:55:16 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 22 Aug 2012 07:55:16 +0200 Subject: [Cython] Jenkins problem In-Reply-To: References: Message-ID: <50347444.4050406@behnel.de> Vitja Makarov, 22.08.2012 06:51: > I've a problem running jenknins job > > https://sage.math.washington.edu:8091/hudson/view/dev-vitek/job/cython-vitek-tests/137/BACKEND=c,PYVERSION=py27/console > > It says "No space left on device". Have something change there? Yes, we're now building in a ramdisk, which is restricted in size to 20GB. Apparently, I had forgotten to add the final cleanup script to your tests job. Should look better now. Stefan From vitja.makarov at gmail.com Wed Aug 22 17:44:32 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 22 Aug 2012 19:44:32 +0400 Subject: [Cython] Jenkins problem In-Reply-To: <50347444.4050406@behnel.de> References: <50347444.4050406@behnel.de> Message-ID: 2012/8/22 Stefan Behnel : > Vitja Makarov, 22.08.2012 06:51: >> I've a problem running jenknins job >> >> https://sage.math.washington.edu:8091/hudson/view/dev-vitek/job/cython-vitek-tests/137/BACKEND=c,PYVERSION=py27/console >> >> It says "No space left on device". Have something change there? > > Yes, we're now building in a ramdisk, which is restricted in size to 20GB. > Apparently, I had forgotten to add the final cleanup script to your tests > job. Should look better now. > > Stefan > Jenkins is down now. -- vitja. From stefan_ml at behnel.de Wed Aug 22 18:01:06 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 22 Aug 2012 18:01:06 +0200 Subject: [Cython] Jenkins problem In-Reply-To: References: <50347444.4050406@behnel.de> Message-ID: <50350242.3000900@behnel.de> Vitja Makarov, 22.08.2012 17:44: > 2012/8/22 Stefan Behnel: >> Vitja Makarov, 22.08.2012 06:51: >>> I've a problem running jenknins job >>> >>> https://sage.math.washington.edu:8091/hudson/view/dev-vitek/job/cython-vitek-tests/137/BACKEND=c,PYVERSION=py27/console >>> >>> It says "No space left on device". Have something change there? >> >> Yes, we're now building in a ramdisk, which is restricted in size to 20GB. >> Apparently, I had forgotten to add the final cleanup script to your tests >> job. Should look better now. > > Jenkins is down now. Yes, the whole sage.math cluster is going through a mental post power-issues crisis. Stefan From stefan_ml at behnel.de Wed Aug 22 19:23:59 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 22 Aug 2012 19:23:59 +0200 Subject: [Cython] Fwd: sagemath cluster In-Reply-To: References: Message-ID: <503515AF.6050703@behnel.de> FYI -------- Original Message -------- Subject: sagemath cluster Date: Wed, 22 Aug 2012 10:21:31 -0700 From: William Stein There were some (unknown-to-me) issue with the power in the server room at University of Washington, resulting from a UPS breaking yesterday. The main fileserver was power cycled and "doesn't work" now, which means that all services hosted on the sage.math cluster will be down until this is fixed (or I switch to using an rsync backup). From vitja.makarov at gmail.com Wed Aug 22 22:11:39 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 23 Aug 2012 00:11:39 +0400 Subject: [Cython] cython-devel-tests-pyregr regression Message-ID: I've found regression: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ -- vitja. From stefan_ml at behnel.de Wed Aug 22 22:25:05 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 22 Aug 2012 22:25:05 +0200 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: Message-ID: <50354021.3040008@behnel.de> Vitja Makarov, 22.08.2012 22:11: > I've found regression: > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ Interesting. It's a Py2 list comprehension in a class body that's failing here: """ class TestHelpSubparsersOrdering(HelpTestCase): subparsers_signatures = [Sig(name=name) for name in ('a', 'b', 'c', 'd', 'e')] """ I wonder why "name" isn't declared as a variable yet at the point where it is being looked up in the function call. Stefan From vitja.makarov at gmail.com Wed Aug 22 22:34:29 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 23 Aug 2012 00:34:29 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: <50354021.3040008@behnel.de> References: <50354021.3040008@behnel.de> Message-ID: 2012/8/23 Stefan Behnel : > Vitja Makarov, 22.08.2012 22:11: >> I've found regression: >> >> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ > > Interesting. It's a Py2 list comprehension in a class body that's failing here: > > """ > class TestHelpSubparsersOrdering(HelpTestCase): > subparsers_signatures = [Sig(name=name) > for name in ('a', 'b', 'c', 'd', 'e')] > """ > > I wonder why "name" isn't declared as a variable yet at the point where it > is being looked up in the function call. > > Stefan > def lookup_relative(self, name, pos): if name == "name": print name from ipdb import set_trace; set_trace() entry = self.lookup_here(name) if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here return entry if self.outer_scope: return self.outer_scope.lookup_relative(name, pos) return None What is that comparison for? -- vitja. From stefan_ml at behnel.de Thu Aug 23 06:47:46 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 06:47:46 +0200 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> Message-ID: <5035B5F2.6040403@behnel.de> Vitja Makarov, 22.08.2012 22:34: > 2012/8/23 Stefan Behnel: >> Vitja Makarov, 22.08.2012 22:11: >>> I've found regression: >>> >>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >> >> Interesting. It's a Py2 list comprehension in a class body that's failing here: >> >> """ >> class TestHelpSubparsersOrdering(HelpTestCase): >> subparsers_signatures = [Sig(name=name) >> for name in ('a', 'b', 'c', 'd', 'e')] >> """ >> >> I wonder why "name" isn't declared as a variable yet at the point where it >> is being looked up in the function call. > > def lookup_relative(self, name, pos): > if name == "name": > print name > from ipdb import set_trace; set_trace() > entry = self.lookup_here(name) > if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here > return entry > if self.outer_scope: > return self.outer_scope.lookup_relative(name, pos) > return None > > > What is that comparison for? Ah, yes, it is wrong in this context. It was meant to prevent names defined further down in the class body from being considered assignments to the name being looked up. Class bodies are not function bodies, assignments in them do not make a name "local". As long as it's not assigned, it's not defined and must be looked up in the outer scope. I think comprehensions are actually the only case where a name is used in the source before its declaration. It should work in all other cases. I had considered solving this problem with the flow control analysis information, but I can't see how that helps me to figure out if an entry is already assigned (i.e. declared) at a given point in the class body. Any idea? Actually, I even wonder if it is a good idea to look up the name directly in the class scope - we may want to inject a local comprehension scope here, as for Py3 comprehensions and genexprs, and just switch the fall-through of the declarations on or off based on the code semantics. Stefan From vitja.makarov at gmail.com Thu Aug 23 07:03:57 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 23 Aug 2012 09:03:57 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: <5035B5F2.6040403@behnel.de> References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> Message-ID: 2012/8/23 Stefan Behnel : > Vitja Makarov, 22.08.2012 22:34: >> 2012/8/23 Stefan Behnel: >>> Vitja Makarov, 22.08.2012 22:11: >>>> I've found regression: >>>> >>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>> >>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>> >>> """ >>> class TestHelpSubparsersOrdering(HelpTestCase): >>> subparsers_signatures = [Sig(name=name) >>> for name in ('a', 'b', 'c', 'd', 'e')] >>> """ >>> >>> I wonder why "name" isn't declared as a variable yet at the point where it >>> is being looked up in the function call. >> >> def lookup_relative(self, name, pos): >> if name == "name": >> print name >> from ipdb import set_trace; set_trace() >> entry = self.lookup_here(name) >> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >> return entry >> if self.outer_scope: >> return self.outer_scope.lookup_relative(name, pos) >> return None >> >> >> What is that comparison for? > > Ah, yes, it is wrong in this context. It was meant to prevent names defined > further down in the class body from being considered assignments to the > name being looked up. Class bodies are not function bodies, assignments in > them do not make a name "local". As long as it's not assigned, it's not > defined and must be looked up in the outer scope. > Do you remember this ticket #671 If there is assignment in the class body we first lookup in the class dict and then in globals. B = 0 def foo(): B = 1 class Foo(): A = B B = B class Bar(): A = B print Foo.A, Foo.B, Bar.A foo() prints "0 0 1" > I think comprehensions are actually the only case where a name is used in > the source before its declaration. It should work in all other cases. > > I had considered solving this problem with the flow control analysis > information, but I can't see how that helps me to figure out if an entry is > already assigned (i.e. declared) at a given point in the class body. > > Any idea? > What would you do with maybe assigned case? > Actually, I even wonder if it is a good idea to look up the name directly > in the class scope - we may want to inject a local comprehension scope > here, as for Py3 comprehensions and genexprs, and just switch the > fall-through of the declarations on or off based on the code semantics. > -- vitja. From stefan_ml at behnel.de Thu Aug 23 07:15:56 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 07:15:56 +0200 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> Message-ID: <5035BC8C.7050403@behnel.de> Vitja Makarov, 23.08.2012 07:03: > 2012/8/23 Stefan Behnel : >> Vitja Makarov, 22.08.2012 22:34: >>> 2012/8/23 Stefan Behnel: >>>> Vitja Makarov, 22.08.2012 22:11: >>>>> I've found regression: >>>>> >>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>> >>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>> >>>> """ >>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>> subparsers_signatures = [Sig(name=name) >>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>> """ >>>> >>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>> is being looked up in the function call. >>> >>> def lookup_relative(self, name, pos): >>> if name == "name": >>> print name >>> from ipdb import set_trace; set_trace() >>> entry = self.lookup_here(name) >>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>> return entry >>> if self.outer_scope: >>> return self.outer_scope.lookup_relative(name, pos) >>> return None >>> >>> >>> What is that comparison for? >> >> Ah, yes, it is wrong in this context. It was meant to prevent names defined >> further down in the class body from being considered assignments to the >> name being looked up. Class bodies are not function bodies, assignments in >> them do not make a name "local". As long as it's not assigned, it's not >> defined and must be looked up in the outer scope. > > Do you remember this ticket #671 > > If there is assignment in the class body we first lookup in the class > dict and then in globals. > > B = 0 > def foo(): > B = 1 > class Foo(): > A = B > B = B > class Bar(): > A = B > print Foo.A, Foo.B, Bar.A > foo() > > prints "0 0 1" In the case at hand, it's not an assignment but a method declaration. Maybe that makes a difference. In any case, this needs some more investigation than I did for my change. I think it can be rolled back completely. >> I think comprehensions are actually the only case where a name is used in >> the source before its declaration. It should work in all other cases. >> >> I had considered solving this problem with the flow control analysis >> information, but I can't see how that helps me to figure out if an entry is >> already assigned (i.e. declared) at a given point in the class body. >> >> Any idea? > > What would you do with maybe assigned case? Hmm, yes - I guess we can't solve the general case at compile time. That's unfortunate, though, because it prevents proper compile time optimisation of builtins in class bodies when their names are assigned at some point, e.g. with a ".type()" method or ".set()", as was the case here. Stefan From vitja.makarov at gmail.com Thu Aug 23 07:32:55 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 23 Aug 2012 09:32:55 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: <5035BC8C.7050403@behnel.de> References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> Message-ID: 2012/8/23 Stefan Behnel : > Vitja Makarov, 23.08.2012 07:03: >> 2012/8/23 Stefan Behnel : >>> Vitja Makarov, 22.08.2012 22:34: >>>> 2012/8/23 Stefan Behnel: >>>>> Vitja Makarov, 22.08.2012 22:11: >>>>>> I've found regression: >>>>>> >>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>>> >>>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>>> >>>>> """ >>>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>>> subparsers_signatures = [Sig(name=name) >>>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>>> """ >>>>> >>>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>>> is being looked up in the function call. >>>> >>>> def lookup_relative(self, name, pos): >>>> if name == "name": >>>> print name >>>> from ipdb import set_trace; set_trace() >>>> entry = self.lookup_here(name) >>>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>>> return entry >>>> if self.outer_scope: >>>> return self.outer_scope.lookup_relative(name, pos) >>>> return None >>>> >>>> >>>> What is that comparison for? >>> >>> Ah, yes, it is wrong in this context. It was meant to prevent names defined >>> further down in the class body from being considered assignments to the >>> name being looked up. Class bodies are not function bodies, assignments in >>> them do not make a name "local". As long as it's not assigned, it's not >>> defined and must be looked up in the outer scope. >> >> Do you remember this ticket #671 >> >> If there is assignment in the class body we first lookup in the class >> dict and then in globals. >> >> B = 0 >> def foo(): >> B = 1 >> class Foo(): >> A = B >> B = B >> class Bar(): >> A = B >> print Foo.A, Foo.B, Bar.A >> foo() >> >> prints "0 0 1" > > In the case at hand, it's not an assignment but a method declaration. Maybe > that makes a difference. > > In any case, this needs some more investigation than I did for my change. I > think it can be rolled back completely. > > >>> I think comprehensions are actually the only case where a name is used in >>> the source before its declaration. It should work in all other cases. >>> >>> I had considered solving this problem with the flow control analysis >>> information, but I can't see how that helps me to figure out if an entry is >>> already assigned (i.e. declared) at a given point in the class body. >>> >>> Any idea? >> >> What would you do with maybe assigned case? > > Hmm, yes - I guess we can't solve the general case at compile time. That's > unfortunate, though, because it prevents proper compile time optimisation > of builtins in class bodies when their names are assigned at some point, > e.g. with a ".type()" method or ".set()", as was the case here. > > Stefan > This is NameNode related problem The following code would fail as well: XXX = 123 class Foo(object): t1 = XXX # XXX cf_is_null = True t2 = XXX # XXX cf_is_null = False XXX = 123 CF assumes that after first apperence of NULL name it must be then set otherwise exception must be raised. NameNode code relays on CF here, so I think it mustn't. We must allways check result of PyObject_GetItem() and if's NULL look at the globals then. I'll try to fix it soon. -- vitja. From vitja.makarov at gmail.com Thu Aug 23 07:39:34 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 23 Aug 2012 09:39:34 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> Message-ID: 2012/8/23 Vitja Makarov : > 2012/8/23 Stefan Behnel : >> Vitja Makarov, 23.08.2012 07:03: >>> 2012/8/23 Stefan Behnel : >>>> Vitja Makarov, 22.08.2012 22:34: >>>>> 2012/8/23 Stefan Behnel: >>>>>> Vitja Makarov, 22.08.2012 22:11: >>>>>>> I've found regression: >>>>>>> >>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>>>> >>>>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>>>> >>>>>> """ >>>>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>>>> subparsers_signatures = [Sig(name=name) >>>>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>>>> """ >>>>>> >>>>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>>>> is being looked up in the function call. >>>>> >>>>> def lookup_relative(self, name, pos): >>>>> if name == "name": >>>>> print name >>>>> from ipdb import set_trace; set_trace() >>>>> entry = self.lookup_here(name) >>>>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>>>> return entry >>>>> if self.outer_scope: >>>>> return self.outer_scope.lookup_relative(name, pos) >>>>> return None >>>>> >>>>> >>>>> What is that comparison for? >>>> >>>> Ah, yes, it is wrong in this context. It was meant to prevent names defined >>>> further down in the class body from being considered assignments to the >>>> name being looked up. Class bodies are not function bodies, assignments in >>>> them do not make a name "local". As long as it's not assigned, it's not >>>> defined and must be looked up in the outer scope. >>> >>> Do you remember this ticket #671 >>> >>> If there is assignment in the class body we first lookup in the class >>> dict and then in globals. >>> >>> B = 0 >>> def foo(): >>> B = 1 >>> class Foo(): >>> A = B >>> B = B >>> class Bar(): >>> A = B >>> print Foo.A, Foo.B, Bar.A >>> foo() >>> >>> prints "0 0 1" >> >> In the case at hand, it's not an assignment but a method declaration. Maybe >> that makes a difference. >> >> In any case, this needs some more investigation than I did for my change. I >> think it can be rolled back completely. >> >> >>>> I think comprehensions are actually the only case where a name is used in >>>> the source before its declaration. It should work in all other cases. >>>> >>>> I had considered solving this problem with the flow control analysis >>>> information, but I can't see how that helps me to figure out if an entry is >>>> already assigned (i.e. declared) at a given point in the class body. >>>> >>>> Any idea? >>> >>> What would you do with maybe assigned case? >> >> Hmm, yes - I guess we can't solve the general case at compile time. That's >> unfortunate, though, because it prevents proper compile time optimisation >> of builtins in class bodies when their names are assigned at some point, >> e.g. with a ".type()" method or ".set()", as was the case here. >> >> Stefan >> > > > > This is NameNode related problem > > The following code would fail as well: > XXX = 123 > class Foo(object): > t1 = XXX # XXX cf_is_null = True > t2 = XXX # XXX cf_is_null = False > XXX = 123 > > CF assumes that after first apperence of NULL name it must be then set > otherwise exception must be raised. > NameNode code relays on CF here, so I think it mustn't. We must > allways check result of PyObject_GetItem() and if's NULL look at the > globals then. > > I'll try to fix it soon. > > I've reverted your commit and here is my fix: https://github.com/vitek/cython/commit/198f254f62360b61c895ba68be0f4dbe07444449 Cause of different class scope lookup rules I think we can't fully trust CF here. -- vitja. From vitja.makarov at gmail.com Thu Aug 23 07:42:05 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 23 Aug 2012 09:42:05 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> Message-ID: 2012/8/23 Vitja Makarov : > 2012/8/23 Vitja Makarov : >> 2012/8/23 Stefan Behnel : >>> Vitja Makarov, 23.08.2012 07:03: >>>> 2012/8/23 Stefan Behnel : >>>>> Vitja Makarov, 22.08.2012 22:34: >>>>>> 2012/8/23 Stefan Behnel: >>>>>>> Vitja Makarov, 22.08.2012 22:11: >>>>>>>> I've found regression: >>>>>>>> >>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>>>>> >>>>>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>>>>> >>>>>>> """ >>>>>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>>>>> subparsers_signatures = [Sig(name=name) >>>>>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>>>>> """ >>>>>>> >>>>>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>>>>> is being looked up in the function call. >>>>>> >>>>>> def lookup_relative(self, name, pos): >>>>>> if name == "name": >>>>>> print name >>>>>> from ipdb import set_trace; set_trace() >>>>>> entry = self.lookup_here(name) >>>>>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>>>>> return entry >>>>>> if self.outer_scope: >>>>>> return self.outer_scope.lookup_relative(name, pos) >>>>>> return None >>>>>> >>>>>> >>>>>> What is that comparison for? >>>>> >>>>> Ah, yes, it is wrong in this context. It was meant to prevent names defined >>>>> further down in the class body from being considered assignments to the >>>>> name being looked up. Class bodies are not function bodies, assignments in >>>>> them do not make a name "local". As long as it's not assigned, it's not >>>>> defined and must be looked up in the outer scope. >>>> >>>> Do you remember this ticket #671 >>>> >>>> If there is assignment in the class body we first lookup in the class >>>> dict and then in globals. >>>> >>>> B = 0 >>>> def foo(): >>>> B = 1 >>>> class Foo(): >>>> A = B >>>> B = B >>>> class Bar(): >>>> A = B >>>> print Foo.A, Foo.B, Bar.A >>>> foo() >>>> >>>> prints "0 0 1" >>> >>> In the case at hand, it's not an assignment but a method declaration. Maybe >>> that makes a difference. >>> >>> In any case, this needs some more investigation than I did for my change. I >>> think it can be rolled back completely. >>> >>> >>>>> I think comprehensions are actually the only case where a name is used in >>>>> the source before its declaration. It should work in all other cases. >>>>> >>>>> I had considered solving this problem with the flow control analysis >>>>> information, but I can't see how that helps me to figure out if an entry is >>>>> already assigned (i.e. declared) at a given point in the class body. >>>>> >>>>> Any idea? >>>> >>>> What would you do with maybe assigned case? >>> >>> Hmm, yes - I guess we can't solve the general case at compile time. That's >>> unfortunate, though, because it prevents proper compile time optimisation >>> of builtins in class bodies when their names are assigned at some point, >>> e.g. with a ".type()" method or ".set()", as was the case here. >>> >>> Stefan >>> >> >> >> >> This is NameNode related problem >> >> The following code would fail as well: >> XXX = 123 >> class Foo(object): >> t1 = XXX # XXX cf_is_null = True >> t2 = XXX # XXX cf_is_null = False >> XXX = 123 >> >> CF assumes that after first apperence of NULL name it must be then set >> otherwise exception must be raised. >> NameNode code relays on CF here, so I think it mustn't. We must >> allways check result of PyObject_GetItem() and if's NULL look at the >> globals then. >> >> I'll try to fix it soon. >> >> > > I've reverted your commit and here is my fix: > https://github.com/vitek/cython/commit/198f254f62360b61c895ba68be0f4dbe07444449 > > Cause of different class scope lookup rules I think we can't fully > trust CF here. > Another (maybe better) solution is to fix CF. It mustn't set cf_maybe_null to False at class scope based on reference success. -- vitja. From stefan_ml at behnel.de Thu Aug 23 09:06:57 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 09:06:57 +0200 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> Message-ID: <5035D691.1030505@behnel.de> Vitja Makarov, 23.08.2012 07:42: > Another (maybe better) solution is to fix CF. It mustn't set > cf_maybe_null to False at class scope based on reference success. Ah, yes - that sounds totally like the right solution. Class scopes are a lot different from function scopes. Stefan From stefan_ml at behnel.de Thu Aug 23 13:03:37 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 13:03:37 +0200 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> Message-ID: <50360E09.7040101@behnel.de> Vitja Makarov, 23.08.2012 07:42: > 2012/8/23 Vitja Makarov: >> 2012/8/23 Vitja Makarov: >>> 2012/8/23 Stefan Behnel: >>>> Vitja Makarov, 23.08.2012 07:03: >>>>> 2012/8/23 Stefan Behnel: >>>>>> Vitja Makarov, 22.08.2012 22:34: >>>>>>> 2012/8/23 Stefan Behnel: >>>>>>>> Vitja Makarov, 22.08.2012 22:11: >>>>>>>>> I've found regression: >>>>>>>>> >>>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>>>>>> >>>>>>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>>>>>> >>>>>>>> """ >>>>>>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>>>>>> subparsers_signatures = [Sig(name=name) >>>>>>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>>>>>> """ >>>>>>>> >>>>>>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>>>>>> is being looked up in the function call. >>>>>>> >>>>>>> def lookup_relative(self, name, pos): >>>>>>> if name == "name": >>>>>>> print name >>>>>>> from ipdb import set_trace; set_trace() >>>>>>> entry = self.lookup_here(name) >>>>>>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>>>>>> return entry >>>>>>> if self.outer_scope: >>>>>>> return self.outer_scope.lookup_relative(name, pos) >>>>>>> return None >>>>>>> >>>>>>> What is that comparison for? >>>>>> >>>>>> Ah, yes, it is wrong in this context. It was meant to prevent names defined >>>>>> further down in the class body from being considered assignments to the >>>>>> name being looked up. Class bodies are not function bodies, assignments in >>>>>> them do not make a name "local". As long as it's not assigned, it's not >>>>>> defined and must be looked up in the outer scope. >>>>> >>>>> Do you remember this ticket #671 >>>>> >>>>> If there is assignment in the class body we first lookup in the class >>>>> dict and then in globals. >>>>> >>>>> B = 0 >>>>> def foo(): >>>>> B = 1 >>>>> class Foo(): >>>>> A = B >>>>> B = B >>>>> class Bar(): >>>>> A = B >>>>> print Foo.A, Foo.B, Bar.A >>>>> foo() >>>>> >>>>> prints "0 0 1" >>>> >>>> In the case at hand, it's not an assignment but a method declaration. Maybe >>>> that makes a difference. >>>> >>>> In any case, this needs some more investigation than I did for my change. I >>>> think it can be rolled back completely. >> >> I've reverted your commit and here is my fix: >> https://github.com/vitek/cython/commit/198f254f62360b61c895ba68be0f4dbe07444449 >> >> Cause of different class scope lookup rules I think we can't fully >> trust CF here. > > Another (maybe better) solution is to fix CF. It mustn't set > cf_maybe_null to False at class scope based on reference success. I've pushed these changes to the master for now. They are correct, even if we decide to improve the control flow decisions here (which would only lead to the same effect anyway). Stefan From vitja.makarov at gmail.com Thu Aug 23 13:06:47 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 23 Aug 2012 15:06:47 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: <50360E09.7040101@behnel.de> References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> <50360E09.7040101@behnel.de> Message-ID: 2012/8/23 Stefan Behnel : > Vitja Makarov, 23.08.2012 07:42: >> 2012/8/23 Vitja Makarov: >>> 2012/8/23 Vitja Makarov: >>>> 2012/8/23 Stefan Behnel: >>>>> Vitja Makarov, 23.08.2012 07:03: >>>>>> 2012/8/23 Stefan Behnel: >>>>>>> Vitja Makarov, 22.08.2012 22:34: >>>>>>>> 2012/8/23 Stefan Behnel: >>>>>>>>> Vitja Makarov, 22.08.2012 22:11: >>>>>>>>>> I've found regression: >>>>>>>>>> >>>>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>>>>>>> >>>>>>>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>>>>>>> >>>>>>>>> """ >>>>>>>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>>>>>>> subparsers_signatures = [Sig(name=name) >>>>>>>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>>>>>>> """ >>>>>>>>> >>>>>>>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>>>>>>> is being looked up in the function call. >>>>>>>> >>>>>>>> def lookup_relative(self, name, pos): >>>>>>>> if name == "name": >>>>>>>> print name >>>>>>>> from ipdb import set_trace; set_trace() >>>>>>>> entry = self.lookup_here(name) >>>>>>>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>>>>>>> return entry >>>>>>>> if self.outer_scope: >>>>>>>> return self.outer_scope.lookup_relative(name, pos) >>>>>>>> return None >>>>>>>> >>>>>>>> What is that comparison for? >>>>>>> >>>>>>> Ah, yes, it is wrong in this context. It was meant to prevent names defined >>>>>>> further down in the class body from being considered assignments to the >>>>>>> name being looked up. Class bodies are not function bodies, assignments in >>>>>>> them do not make a name "local". As long as it's not assigned, it's not >>>>>>> defined and must be looked up in the outer scope. >>>>>> >>>>>> Do you remember this ticket #671 >>>>>> >>>>>> If there is assignment in the class body we first lookup in the class >>>>>> dict and then in globals. >>>>>> >>>>>> B = 0 >>>>>> def foo(): >>>>>> B = 1 >>>>>> class Foo(): >>>>>> A = B >>>>>> B = B >>>>>> class Bar(): >>>>>> A = B >>>>>> print Foo.A, Foo.B, Bar.A >>>>>> foo() >>>>>> >>>>>> prints "0 0 1" >>>>> >>>>> In the case at hand, it's not an assignment but a method declaration. Maybe >>>>> that makes a difference. >>>>> >>>>> In any case, this needs some more investigation than I did for my change. I >>>>> think it can be rolled back completely. >>> >>> I've reverted your commit and here is my fix: >>> https://github.com/vitek/cython/commit/198f254f62360b61c895ba68be0f4dbe07444449 >>> >>> Cause of different class scope lookup rules I think we can't fully >>> trust CF here. >> >> Another (maybe better) solution is to fix CF. It mustn't set >> cf_maybe_null to False at class scope based on reference success. > > I've pushed these changes to the master for now. They are correct, even if > we decide to improve the control flow decisions here (which would only lead > to the same effect anyway). > Ok, I'll take a look later. -- vitja. From stefan_ml at behnel.de Thu Aug 23 20:13:19 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 20:13:19 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate Message-ID: <503672BF.5070702@behnel.de> Hello everyone, on behalf of the Cython project team, I'm proud to announce the release of our third beta of Cython 0.17. This is our first and hopefully last release candidate for 0.17, another major step forward in the development of the language that will make life easier for a lot of users, rounds up some rough edges of the compiler and adds (preliminary) support for CPython 3.3 and PyPy. Download: http://cython.org/release/Cython-0.17b3.tar.gz Release notes: http://wiki.cython.org/ReleaseNotes-0.17 Documentation: http://docs.cython.org/ Major features of this release include: * vastly improved integration with the C++ STL containers http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library http://docs.cython.org/src/tutorial/strings.html#c-strings * "yield from" delegation between generators (PEP 380) http://www.python.org/dev/peps/pep-0380/ * alpha quality support for PyPy (via cpyext) http://docs.cython.org/src/userguide/pypy.html Several other features and improvements are listed in the release notes: http://wiki.cython.org/ReleaseNotes-0.17 Have fun, Stefan From lists at onerussian.com Thu Aug 23 21:23:32 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Thu, 23 Aug 2012 15:23:32 -0400 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503672BF.5070702@behnel.de> References: <503672BF.5070702@behnel.de> Message-ID: <20120823192332.GX5866@onerussian.com> congrats! my not so wild but still a guess that 0.17b3 corresponds to 5b0e8f8b96e353ac6d12d54f86babee9ff6f1bf8 git commit (never know for sure if there is no tag :-P )? ;) On Thu, 23 Aug 2012, Stefan Behnel wrote: > Hello everyone, > on behalf of the Cython project team, I'm proud to announce the release of > our third beta of Cython 0.17. This is our first and hopefully last release > candidate for 0.17, another major step forward in the development of the > language that will make life easier for a lot of users, rounds up some > rough edges of the compiler and adds (preliminary) support for CPython 3.3 > and PyPy. -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From stefan_ml at behnel.de Thu Aug 23 21:55:21 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 23 Aug 2012 21:55:21 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <20120823192332.GX5866@onerussian.com> References: <503672BF.5070702@behnel.de> <20120823192332.GX5866@onerussian.com> Message-ID: <50368AA9.4040401@behnel.de> Yaroslav Halchenko, 23.08.2012 21:23: > my not so wild but still a guess that 0.17b3 corresponds to > 5b0e8f8b96e353ac6d12d54f86babee9ff6f1bf8 git commit (never know for sure > if there is no tag :-P )? ;) http://wiki.cython.org/ReleaseNotes-0.17 hg-git isn't all that good at mirroring tags back into github. I'll eventually add them with git, unless someone beats me to it. Stefan From vitja.makarov at gmail.com Thu Aug 23 22:13:19 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 24 Aug 2012 00:13:19 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> <50360E09.7040101@behnel.de> Message-ID: 2012/8/23 Vitja Makarov : > 2012/8/23 Stefan Behnel : >> Vitja Makarov, 23.08.2012 07:42: >>> 2012/8/23 Vitja Makarov: >>>> 2012/8/23 Vitja Makarov: >>>>> 2012/8/23 Stefan Behnel: >>>>>> Vitja Makarov, 23.08.2012 07:03: >>>>>>> 2012/8/23 Stefan Behnel: >>>>>>>> Vitja Makarov, 22.08.2012 22:34: >>>>>>>>> 2012/8/23 Stefan Behnel: >>>>>>>>>> Vitja Makarov, 22.08.2012 22:11: >>>>>>>>>>> I've found regression: >>>>>>>>>>> >>>>>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>>>>>>>> >>>>>>>>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>>>>>>>> >>>>>>>>>> """ >>>>>>>>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>>>>>>>> subparsers_signatures = [Sig(name=name) >>>>>>>>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>>>>>>>> """ >>>>>>>>>> >>>>>>>>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>>>>>>>> is being looked up in the function call. >>>>>>>>> >>>>>>>>> def lookup_relative(self, name, pos): >>>>>>>>> if name == "name": >>>>>>>>> print name >>>>>>>>> from ipdb import set_trace; set_trace() >>>>>>>>> entry = self.lookup_here(name) >>>>>>>>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>>>>>>>> return entry >>>>>>>>> if self.outer_scope: >>>>>>>>> return self.outer_scope.lookup_relative(name, pos) >>>>>>>>> return None >>>>>>>>> >>>>>>>>> What is that comparison for? >>>>>>>> >>>>>>>> Ah, yes, it is wrong in this context. It was meant to prevent names defined >>>>>>>> further down in the class body from being considered assignments to the >>>>>>>> name being looked up. Class bodies are not function bodies, assignments in >>>>>>>> them do not make a name "local". As long as it's not assigned, it's not >>>>>>>> defined and must be looked up in the outer scope. >>>>>>> >>>>>>> Do you remember this ticket #671 >>>>>>> >>>>>>> If there is assignment in the class body we first lookup in the class >>>>>>> dict and then in globals. >>>>>>> >>>>>>> B = 0 >>>>>>> def foo(): >>>>>>> B = 1 >>>>>>> class Foo(): >>>>>>> A = B >>>>>>> B = B >>>>>>> class Bar(): >>>>>>> A = B >>>>>>> print Foo.A, Foo.B, Bar.A >>>>>>> foo() >>>>>>> >>>>>>> prints "0 0 1" >>>>>> >>>>>> In the case at hand, it's not an assignment but a method declaration. Maybe >>>>>> that makes a difference. >>>>>> >>>>>> In any case, this needs some more investigation than I did for my change. I >>>>>> think it can be rolled back completely. >>>> >>>> I've reverted your commit and here is my fix: >>>> https://github.com/vitek/cython/commit/198f254f62360b61c895ba68be0f4dbe07444449 >>>> >>>> Cause of different class scope lookup rules I think we can't fully >>>> trust CF here. >>> >>> Another (maybe better) solution is to fix CF. It mustn't set >>> cf_maybe_null to False at class scope based on reference success. >> >> I've pushed these changes to the master for now. They are correct, even if >> we decide to improve the control flow decisions here (which would only lead >> to the same effect anyway). >> > > Ok, I'll take a look later. > Here is fix from CF point of view: https://github.com/vitek/cython/commit/cbd676c598a7ea0ba61262d46c1c733d362c68da Let's wait for jenkins now. -- vitja. From vitja.makarov at gmail.com Fri Aug 24 06:31:58 2012 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 24 Aug 2012 08:31:58 +0400 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> <50360E09.7040101@behnel.de> Message-ID: 2012/8/24 Vitja Makarov : > 2012/8/23 Vitja Makarov : >> 2012/8/23 Stefan Behnel : >>> Vitja Makarov, 23.08.2012 07:42: >>>> 2012/8/23 Vitja Makarov: >>>>> 2012/8/23 Vitja Makarov: >>>>>> 2012/8/23 Stefan Behnel: >>>>>>> Vitja Makarov, 23.08.2012 07:03: >>>>>>>> 2012/8/23 Stefan Behnel: >>>>>>>>> Vitja Makarov, 22.08.2012 22:34: >>>>>>>>>> 2012/8/23 Stefan Behnel: >>>>>>>>>>> Vitja Makarov, 22.08.2012 22:11: >>>>>>>>>>>> I've found regression: >>>>>>>>>>>> >>>>>>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr/ >>>>>>>>>>> >>>>>>>>>>> Interesting. It's a Py2 list comprehension in a class body that's failing here: >>>>>>>>>>> >>>>>>>>>>> """ >>>>>>>>>>> class TestHelpSubparsersOrdering(HelpTestCase): >>>>>>>>>>> subparsers_signatures = [Sig(name=name) >>>>>>>>>>> for name in ('a', 'b', 'c', 'd', 'e')] >>>>>>>>>>> """ >>>>>>>>>>> >>>>>>>>>>> I wonder why "name" isn't declared as a variable yet at the point where it >>>>>>>>>>> is being looked up in the function call. >>>>>>>>>> >>>>>>>>>> def lookup_relative(self, name, pos): >>>>>>>>>> if name == "name": >>>>>>>>>> print name >>>>>>>>>> from ipdb import set_trace; set_trace() >>>>>>>>>> entry = self.lookup_here(name) >>>>>>>>>> if entry is not None and entry.pos[1:] <= pos[1:]: # Lookup fails here >>>>>>>>>> return entry >>>>>>>>>> if self.outer_scope: >>>>>>>>>> return self.outer_scope.lookup_relative(name, pos) >>>>>>>>>> return None >>>>>>>>>> >>>>>>>>>> What is that comparison for? >>>>>>>>> >>>>>>>>> Ah, yes, it is wrong in this context. It was meant to prevent names defined >>>>>>>>> further down in the class body from being considered assignments to the >>>>>>>>> name being looked up. Class bodies are not function bodies, assignments in >>>>>>>>> them do not make a name "local". As long as it's not assigned, it's not >>>>>>>>> defined and must be looked up in the outer scope. >>>>>>>> >>>>>>>> Do you remember this ticket #671 >>>>>>>> >>>>>>>> If there is assignment in the class body we first lookup in the class >>>>>>>> dict and then in globals. >>>>>>>> >>>>>>>> B = 0 >>>>>>>> def foo(): >>>>>>>> B = 1 >>>>>>>> class Foo(): >>>>>>>> A = B >>>>>>>> B = B >>>>>>>> class Bar(): >>>>>>>> A = B >>>>>>>> print Foo.A, Foo.B, Bar.A >>>>>>>> foo() >>>>>>>> >>>>>>>> prints "0 0 1" >>>>>>> >>>>>>> In the case at hand, it's not an assignment but a method declaration. Maybe >>>>>>> that makes a difference. >>>>>>> >>>>>>> In any case, this needs some more investigation than I did for my change. I >>>>>>> think it can be rolled back completely. >>>>> >>>>> I've reverted your commit and here is my fix: >>>>> https://github.com/vitek/cython/commit/198f254f62360b61c895ba68be0f4dbe07444449 >>>>> >>>>> Cause of different class scope lookup rules I think we can't fully >>>>> trust CF here. >>>> >>>> Another (maybe better) solution is to fix CF. It mustn't set >>>> cf_maybe_null to False at class scope based on reference success. >>> >>> I've pushed these changes to the master for now. They are correct, even if >>> we decide to improve the control flow decisions here (which would only lead >>> to the same effect anyway). >>> >> >> Ok, I'll take a look later. >> > > Here is fix from CF point of view: > https://github.com/vitek/cython/commit/cbd676c598a7ea0ba61262d46c1c733d362c68da > > Let's wait for jenkins now. > Jenkins still is not happy about upstream/master. It shows 49.2K instead of previous 50.5K -- vitja. From cgohlke at uci.edu Fri Aug 24 07:20:47 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 23 Aug 2012 22:20:47 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503672BF.5070702@behnel.de> References: <503672BF.5070702@behnel.de> Message-ID: <50370F2F.4090503@uci.edu> On 8/23/2012 11:13 AM, Stefan Behnel wrote: > Hello everyone, > > on behalf of the Cython project team, I'm proud to announce the release of > our third beta of Cython 0.17. This is our first and hopefully last release > candidate for 0.17, another major step forward in the development of the > language that will make life easier for a lot of users, rounds up some > rough edges of the compiler and adds (preliminary) support for CPython 3.3 > and PyPy. > > Download: http://cython.org/release/Cython-0.17b3.tar.gz > > Release notes: http://wiki.cython.org/ReleaseNotes-0.17 > > Documentation: http://docs.cython.org/ > > > Major features of this release include: > > * vastly improved integration with the C++ STL containers > > http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library > > http://docs.cython.org/src/tutorial/strings.html#c-strings > > * "yield from" delegation between generators (PEP 380) > > http://www.python.org/dev/peps/pep-0380/ > > * alpha quality support for PyPy (via cpyext) > > http://docs.cython.org/src/userguide/pypy.html > > > Several other features and improvements are listed in the release notes: > > http://wiki.cython.org/ReleaseNotes-0.17 > > > Have fun, > > Stefan Hello Stefan, Thank you. I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. 32 bit Python 2.7 works well, only 4 test failures. On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes during `test_slice_assignment (memslice.__test__)`. I tested two computers. The Windows executive can not identify in which specific module it crashes, and neither enabling faulthandler nor building with debug symbols gives any useful information. Can anyone reproduce this? It seems compiler specific since Python 3.3, which is using msvc10, does not crash. When disabling test_slice_assignment, runtests.py completes with many failures. The results of `runtests.py -v -v` are at . Christoph From stefan_ml at behnel.de Fri Aug 24 15:48:22 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 24 Aug 2012 15:48:22 +0200 Subject: [Cython] cython-devel-tests-pyregr regression In-Reply-To: References: <50354021.3040008@behnel.de> <5035B5F2.6040403@behnel.de> <5035BC8C.7050403@behnel.de> <50360E09.7040101@behnel.de> Message-ID: <50378626.8090001@behnel.de> Vitja Makarov, 24.08.2012 06:31: > Jenkins still is not happy about upstream/master. > > It shows 49.2K instead of previous 50.5K Part of the difference should be that I disabled the ctypes test. It crashes in the pyregr test runs that pyximport the stdlib modules. I don't see anything obvious that I would consider a degradation during the last few runs. Stefan From markflorisson88 at gmail.com Fri Aug 24 20:40:40 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 24 Aug 2012 19:40:40 +0100 Subject: [Cython] array expressions Message-ID: Hey, Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 It includes the IndexNode refactoring branch as well. This has been the work this last summer for the gsoc, with great supervision from Dag, who helped steer the project in a great direction to make it reusable (it's partially included in Numba and will likely be in Theano in the future, hopefully others as well). I also wrote a thesis for my master's, which can be found here https://github.com/markflorisson88/minivect/tree/master/thesis, which can shed some light on some parts of the design and performance aspects. Performance graphs can also be found here: https://github.com/markflorisson88/minivect/tree/master/bench/graphs So anyway, how would you prefer dealing with the minivect submodule? We could include it verbatim, with any modifications made to minivect directly, since we'd have separate git histories. We could alternatively make it an optional submodule which is only required when actually using array expressions. I like the latter, but anything is fine with me really. Mark From stefan_ml at behnel.de Fri Aug 24 21:43:12 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 24 Aug 2012 21:43:12 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <50370F2F.4090503@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> Message-ID: <5037D950.1010106@behnel.de> Hi, thanks for testing! Christoph Gohlke, 24.08.2012 07:20: > I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. > > 32 bit Python 2.7 works well, only 4 test failures. Three of those errors are in OpenMP tests - is OpenMP supported in your build environment? The other one is the new "initial_file_path" test that fails with this linker error: """ C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG /LIBPATH:X:\Python27\libs /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ build\temp.win32-2.7\Release\my_test_package\__init__.obj /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest LINK : error LNK2001: unresolved external symbol init__init__ """ Maybe the Windows build of distutils is broken here - it seems to assume the wrong module name for the package module. I guess compiling package modules is just an overall badly supported feature in CPython... > On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes during > `test_slice_assignment (memslice.__test__)`. I tested two computers. The > Windows executive can not identify in which specific module it crashes, and > neither enabling faulthandler nor building with debug symbols gives any > useful information. Can anyone reproduce this? It seems compiler specific > since Python 3.3, which is using msvc10, does not crash. Hmm, yes, sounds like a problem with the compiler. Would be good to get this sorted out, but it's almost impossible to debug something like this from a distance. > When disabling > test_slice_assignment, runtests.py completes with many failures. > > The results of `runtests.py -v -v` are at > . The 64bit output looks so broken that I wonder what went wrong here. I mean, most of the problems look like this: """ Expected: Traceback (most recent call last): TypeError: m() takes at most 2 positional arguments (3 given) Got: Traceback (most recent call last): TypeError: m() takes at most %Id positional argument%s (%Id given) """ I have no idea how that can happen. I can see two other problems, one is the linker warning about the module init function in Py3: """ bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified multiple times; using first specification """ The other one is about mixing Py_ssize_t and int (and some other) types all over the place: """ bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to 'int', possible loss of data """ Some of them look like we'd need an explicit cast in the C code somewhere, others might hint at lax type usage in tests. There's also this: """ C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\xlocale(323) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify /EHsc """ Stefan From cgohlke at uci.edu Sat Aug 25 04:07:50 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Fri, 24 Aug 2012 19:07:50 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <5037D950.1010106@behnel.de> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> Message-ID: <50383376.7090500@uci.edu> Hi, On 8/24/2012 12:43 PM, Stefan Behnel wrote: > Hi, > > thanks for testing! > > Christoph Gohlke, 24.08.2012 07:20: >> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >> >> 32 bit Python 2.7 works well, only 4 test failures. > > Three of those errors are in OpenMP tests - is OpenMP supported in your > build environment? OpenMP is available on my system, and parallel.pyd is linked to the openmp library. The prange tests fail only sometimes. On my system, the prange index is sometimes left at the start (zero) of the range, while the tests expect the index to be left at the stop of the range. According to the Cython prange enhancements webpage "the iterations of the loop body can be executed in any order" . Where does that leave the loop index? > > The other one is the new "initial_file_path" test that fails with this > linker error: > > """ > C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL > /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG /LIBPATH:X:\Python27\libs > /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ > build\temp.win32-2.7\Release\my_test_package\__init__.obj > /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib > /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest > > LINK : error LNK2001: unresolved external symbol init__init__ > """ > > Maybe the Windows build of distutils is broken here - it seems to assume > the wrong module name for the package module. I think this is an issue with the test. The extension does compile and link outside of the tests. > > I guess compiling package modules is just an overall badly supported > feature in CPython... > > >> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes during >> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >> Windows executive can not identify in which specific module it crashes, and >> neither enabling faulthandler nor building with debug symbols gives any >> useful information. Can anyone reproduce this? It seems compiler specific >> since Python 3.3, which is using msvc10, does not crash. > > Hmm, yes, sounds like a problem with the compiler. Would be good to get > this sorted out, but it's almost impossible to debug something like this > from a distance. Maybe the following simple example is related. It fails (not crash) when compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 and 64 bit): ``` from cython.view cimport array as cvarray import numpy as np narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) cdef int[:, :, ::1] a = narr cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] print narr[2:8:2, -4:1:-1, 1:3].shape print b.shape[0], b.shape[1], b.shape[2] ``` On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) > > >> When disabling >> test_slice_assignment, runtests.py completes with many failures. >> >> The results of `runtests.py -v -v` are at >> . > > The 64bit output looks so broken that I wonder what went wrong here. I > mean, most of the problems look like this: > > """ > Expected: > Traceback (most recent call last): > TypeError: m() takes at most 2 positional arguments (3 given) > Got: > Traceback (most recent call last): > TypeError: m() takes at most %Id positional argument%s (%Id given) > """ > > I have no idea how that can happen. > > I can see two other problems, one is the linker warning about the module > init function in Py3: > > """ > bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified > multiple times; using first specification > """ This is "normal". > > The other one is about mixing Py_ssize_t and int (and some other) types all > over the place: > > """ > bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to > 'int', possible loss of data > """ > > Some of them look like we'd need an explicit cast in the C code somewhere, > others might hint at lax type usage in tests. > > There's also this: > > """ > C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\xlocale(323) > : warning C4530: C++ exception handler used, but unwind semantics are not > enabled. Specify /EHsc > """ > > Stefan > Thanks, Christoph From stefan_ml at behnel.de Sat Aug 25 15:51:22 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 25 Aug 2012 15:51:22 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <50383376.7090500@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> Message-ID: <5038D85A.6090505@behnel.de> Christoph Gohlke, 25.08.2012 04:07: > On 8/24/2012 12:43 PM, Stefan Behnel wrote: >> Christoph Gohlke, 24.08.2012 07:20: >>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>> >>> 32 bit Python 2.7 works well, only 4 test failures. >> >> Three of those errors are in OpenMP tests - is OpenMP supported in your >> build environment? > > OpenMP is available on my system, and parallel.pyd is linked to the openmp > library. The prange tests fail only sometimes. On my system, the prange > index is sometimes left at the start (zero) of the range, while the tests > expect the index to be left at the stop of the range. According to the > Cython prange enhancements webpage "the iterations of the loop body can be > executed in any order" . Where > does that leave the loop index? Mark should know best. I agree that this seems a bit smelly. >> The other one is the new "initial_file_path" test that fails with this >> linker error: >> >> """ >> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG /LIBPATH:X:\Python27\libs >> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >> build\temp.win32-2.7\Release\my_test_package\__init__.obj >> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >> >> >> LINK : error LNK2001: unresolved external symbol init__init__ >> """ >> >> Maybe the Windows build of distutils is broken here - it seems to assume >> the wrong module name for the package module. > > I think this is an issue with the test. The extension does compile and link > outside of the tests. Hmm, did you run the setup.py script? It compiles the "__init__.py" file as well, which is the problem above. Could you check if that's properly being imported as a shared library? (The way "somefile.srctree" tests work is that they explode into a directory structure and then just run the commands in the first lines.) >>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes during >>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>> Windows executive can not identify in which specific module it crashes, and >>> neither enabling faulthandler nor building with debug symbols gives any >>> useful information. Can anyone reproduce this? It seems compiler specific >>> since Python 3.3, which is using msvc10, does not crash. >> >> Hmm, yes, sounds like a problem with the compiler. Would be good to get >> this sorted out, but it's almost impossible to debug something like this >> from a distance. > > Maybe the following simple example is related. It fails (not crash) when > compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 > and 64 bit): > > ``` > from cython.view cimport array as cvarray > import numpy as np > > narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) > > cdef int[:, :, ::1] a = narr > cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] > > print narr[2:8:2, -4:1:-1, 1:3].shape > print b.shape[0], b.shape[1], b.shape[2] > ``` > > On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) I'll leave that for the others to comment. >> """ >> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >> multiple times; using first specification >> """ > > This is "normal". So you have an idea why this happens? Is it because of that explicit /EXPORT command line switch for the module init function name? Stefan From cgohlke at uci.edu Sun Aug 26 10:03:40 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 26 Aug 2012 01:03:40 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <5038D85A.6090505@behnel.de> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <5038D85A.6090505@behnel.de> Message-ID: <5039D85C.1090600@uci.edu> On 8/25/2012 6:51 AM, Stefan Behnel wrote: > Christoph Gohlke, 25.08.2012 04:07: >> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>> Christoph Gohlke, 24.08.2012 07:20: >>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>> >>>> 32 bit Python 2.7 works well, only 4 test failures. >>> >>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>> build environment? >> >> OpenMP is available on my system, and parallel.pyd is linked to the openmp >> library. The prange tests fail only sometimes. On my system, the prange >> index is sometimes left at the start (zero) of the range, while the tests >> expect the index to be left at the stop of the range. According to the >> Cython prange enhancements webpage "the iterations of the loop body can be >> executed in any order" . Where >> does that leave the loop index? > > Mark should know best. I agree that this seems a bit smelly. > > >>> The other one is the new "initial_file_path" test that fails with this >>> linker error: >>> >>> """ >>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG /LIBPATH:X:\Python27\libs >>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>> >>> >>> LINK : error LNK2001: unresolved external symbol init__init__ >>> """ >>> >>> Maybe the Windows build of distutils is broken here - it seems to assume >>> the wrong module name for the package module. >> >> I think this is an issue with the test. The extension does compile and link >> outside of the tests. > > Hmm, did you run the setup.py script? It compiles the "__init__.py" file as > well, which is the problem above. Could you check if that's properly being > imported as a shared library? > > (The way "somefile.srctree" tests work is that they explode into a > directory structure and then just run the commands in the first lines.) > > >>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes during >>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>>> Windows executive can not identify in which specific module it crashes, and >>>> neither enabling faulthandler nor building with debug symbols gives any >>>> useful information. Can anyone reproduce this? It seems compiler specific >>>> since Python 3.3, which is using msvc10, does not crash. >>> >>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>> this sorted out, but it's almost impossible to debug something like this >>> from a distance. >> >> Maybe the following simple example is related. It fails (not crash) when >> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 >> and 64 bit): >> >> ``` >> from cython.view cimport array as cvarray >> import numpy as np >> >> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >> >> cdef int[:, :, ::1] a = narr >> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >> >> print narr[2:8:2, -4:1:-1, 1:3].shape >> print b.shape[0], b.shape[1], b.shape[2] >> ``` >> >> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) > > I'll leave that for the others to comment. This failure and the test crash disappear when auto-inlining is disabled, e.g. using the `/Ob1` compiler switch Is there a way to add a C pre-processor `#pragma` to a Cython generated C file? I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` but the pragma ends up inside the Python module init function, which fails to compile with `error C2156: pragma must be outside function`. > > >>> """ >>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>> multiple times; using first specification >>> """ >> >> This is "normal". > > So you have an idea why this happens? Is it because of that explicit > /EXPORT command line switch for the module init function name? seems to be an issue with distutils: Christoph > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > From stefan_ml at behnel.de Sun Aug 26 11:36:44 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 26 Aug 2012 11:36:44 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <5039D85C.1090600@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <5038D85A.6090505@behnel.de> <5039D85C.1090600@uci.edu> Message-ID: <5039EE2C.3080100@behnel.de> Christoph Gohlke, 26.08.2012 10:03: > On 8/25/2012 6:51 AM, Stefan Behnel wrote: >> Christoph Gohlke, 25.08.2012 04:07: >>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>> Christoph Gohlke, 24.08.2012 07:20: >>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>> during >>>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>>>> Windows executive can not identify in which specific module it >>>>> crashes, and >>>>> neither enabling faulthandler nor building with debug symbols gives any >>>>> useful information. Can anyone reproduce this? It seems compiler specific >>>>> since Python 3.3, which is using msvc10, does not crash. >>>> >>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>> this sorted out, but it's almost impossible to debug something like this >>>> from a distance. >>> >>> Maybe the following simple example is related. It fails (not crash) when >>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 >>> and 64 bit): >>> >>> ``` >>> from cython.view cimport array as cvarray >>> import numpy as np >>> >>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>> >>> cdef int[:, :, ::1] a = narr >>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>> >>> print narr[2:8:2, -4:1:-1, 1:3].shape >>> print b.shape[0], b.shape[1], b.shape[2] >>> ``` >>> >>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >> >> I'll leave that for the others to comment. > > This failure and the test crash disappear when auto-inlining is disabled, > e.g. using the `/Ob1` compiler switch > Thanks for figuring that out. > Is there a way to add a C pre-processor `#pragma` to a Cython generated C > file? I don't think so, not in arbitrary places. > I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` but > the pragma ends up inside the Python module init function, which fails to > compile with `error C2156: pragma must be outside function`. As a quick fix, we could wrap (or just start) our utility code section with that pragma. It starts after all user code. According to the docs, the pragma only disables automatic inlining in the specific range, not explicitly requested inlining. http://msdn.microsoft.com/en-us/library/ah08w5c3%28v=vs.71%29.aspx We usually know what we consider worth inlining and what doesn't need to be. If we can find out what exact functions trigger this, we can start being more specific about where to enable the pragma, but I think it's ok to enable it on all utility code for now. Could you figure out a suitable preprocessor guard that only enables it for the affected compilers? Stefan From stefan_ml at behnel.de Sun Aug 26 11:41:39 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 26 Aug 2012 11:41:39 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <5039EE2C.3080100@behnel.de> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <5038D85A.6090505@behnel.de> <5039D85C.1090600@uci.edu> <5039EE2C.3080100@behnel.de> Message-ID: <5039EF53.50806@behnel.de> Stefan Behnel, 26.08.2012 11:36: > Christoph Gohlke, 26.08.2012 10:03: >> On 8/25/2012 6:51 AM, Stefan Behnel wrote: >>> Christoph Gohlke, 25.08.2012 04:07: >>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>>> Christoph Gohlke, 24.08.2012 07:20: >>>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>>> during >>>>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>>>>> Windows executive can not identify in which specific module it >>>>>> crashes, and >>>>>> neither enabling faulthandler nor building with debug symbols gives any >>>>>> useful information. Can anyone reproduce this? It seems compiler specific >>>>>> since Python 3.3, which is using msvc10, does not crash. >>>>> >>>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>>> this sorted out, but it's almost impossible to debug something like this >>>>> from a distance. >>>> >>>> Maybe the following simple example is related. It fails (not crash) when >>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 >>>> and 64 bit): >>>> >>>> ``` >>>> from cython.view cimport array as cvarray >>>> import numpy as np >>>> >>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>>> >>>> cdef int[:, :, ::1] a = narr >>>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>>> >>>> print narr[2:8:2, -4:1:-1, 1:3].shape >>>> print b.shape[0], b.shape[1], b.shape[2] >>>> ``` >>>> >>>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >>> >>> I'll leave that for the others to comment. >> >> This failure and the test crash disappear when auto-inlining is disabled, >> e.g. using the `/Ob1` compiler switch >> > > Thanks for figuring that out. > > >> Is there a way to add a C pre-processor `#pragma` to a Cython generated C >> file? > > I don't think so, not in arbitrary places. > > >> I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` but >> the pragma ends up inside the Python module init function, which fails to >> compile with `error C2156: pragma must be outside function`. > > As a quick fix, we could wrap (or just start) our utility code section with > that pragma. It starts after all user code. According to the docs, the > pragma only disables automatic inlining in the specific range, not > explicitly requested inlining. > > http://msdn.microsoft.com/en-us/library/ah08w5c3%28v=vs.71%29.aspx > > We usually know what we consider worth inlining and what doesn't need to > be. If we can find out what exact functions trigger this, we can start > being more specific about where to enable the pragma, but I think it's ok > to enable it on all utility code for now. > > Could you figure out a suitable preprocessor guard that only enables it for > the affected compilers? A patch would essentially look like this: diff -r 18fed0dec20e Cython/Compiler/Code.py --- a/Cython/Compiler/Code.py Sun Aug 26 00:54:01 2012 +0200 +++ b/Cython/Compiler/Code.py Sun Aug 26 11:39:29 2012 +0200 @@ -893,6 +893,8 @@ code.write('\n#line 1 "cython_utility"\n') code.putln("") code.putln("/* Runtime support code */") + code.putln("#pragma auto_inline(off)") + code.putln("") def finalize_main_c_code(self): self.close_global_decls() Stefan From markflorisson88 at gmail.com Sun Aug 26 13:08:39 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 26 Aug 2012 12:08:39 +0100 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <50383376.7090500@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> Message-ID: On 25 August 2012 03:07, Christoph Gohlke wrote: > Hi, > > > On 8/24/2012 12:43 PM, Stefan Behnel wrote: >> >> Hi, >> >> thanks for testing! >> >> Christoph Gohlke, 24.08.2012 07:20: >>> >>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>> >>> 32 bit Python 2.7 works well, only 4 test failures. >> >> >> Three of those errors are in OpenMP tests - is OpenMP supported in your >> build environment? > > > OpenMP is available on my system, and parallel.pyd is linked to the openmp > library. The prange tests fail only sometimes. On my system, the prange > index is sometimes left at the start (zero) of the range, while the tests > expect the index to be left at the stop of the range. According to the > Cython prange enhancements webpage "the iterations of the loop body can be > executed in any order" . Where > does that leave the loop index? > I should be the value from the last iteration, but in my experience many compilers have buggy OpenMP implementations. I think your compiler doesn't correctly support the lastprivate clause in all situations. For instance, test_prange fails (which doesn't have any break/return/exceptions), which simply has a lastprivate clause and a schedule clause set to 'dynamic'. The test without the schedule clause works fine (or maybe it's just luck). Or maybe it doesn't support multiple lastprivate() clauses? I'm not entirely sure... It also seems the thread limit on your system is 1. In any case, the generated code for these tests looks correct to me, but we've had similar problems before with different compilers... > >> >> The other one is the new "initial_file_path" test that fails with this >> linker error: >> >> """ >> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >> /LIBPATH:X:\Python27\libs >> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >> build\temp.win32-2.7\Release\my_test_package\__init__.obj >> >> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >> >> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >> >> LINK : error LNK2001: unresolved external symbol init__init__ >> """ >> >> Maybe the Windows build of distutils is broken here - it seems to assume >> the wrong module name for the package module. > > > I think this is an issue with the test. The extension does compile and link > outside of the tests. > > > >> >> I guess compiling package modules is just an overall badly supported >> feature in CPython... >> >> >>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>> during >>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>> Windows executive can not identify in which specific module it crashes, >>> and >>> neither enabling faulthandler nor building with debug symbols gives any >>> useful information. Can anyone reproduce this? It seems compiler specific >>> since Python 3.3, which is using msvc10, does not crash. >> >> >> Hmm, yes, sounds like a problem with the compiler. Would be good to get >> this sorted out, but it's almost impossible to debug something like this >> from a distance. > > > > Maybe the following simple example is related. It fails (not crash) when > compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 > and 64 bit): > > ``` > from cython.view cimport array as cvarray > import numpy as np > > narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) > > cdef int[:, :, ::1] a = narr > cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] > > print narr[2:8:2, -4:1:-1, 1:3].shape > print b.shape[0], b.shape[1], b.shape[2] > ``` > > On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) > > > >> >> >>> When disabling >>> test_slice_assignment, runtests.py completes with many failures. >>> >>> The results of `runtests.py -v -v` are at >>> . >> >> >> The 64bit output looks so broken that I wonder what went wrong here. I >> mean, most of the problems look like this: >> >> """ >> Expected: >> Traceback (most recent call last): >> TypeError: m() takes at most 2 positional arguments (3 given) >> Got: >> Traceback (most recent call last): >> TypeError: m() takes at most %Id positional argument%s (%Id given) >> """ >> >> I have no idea how that can happen. >> >> I can see two other problems, one is the linker warning about the module >> init function in Py3: >> >> """ >> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >> multiple times; using first specification >> """ > > > This is "normal". > > > >> >> The other one is about mixing Py_ssize_t and int (and some other) types >> all >> over the place: >> >> """ >> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >> 'int', possible loss of data >> """ >> >> Some of them look like we'd need an explicit cast in the C code somewhere, >> others might hint at lax type usage in tests. >> >> There's also this: >> >> """ >> C:\Program Files (x86)\Microsoft Visual Studio >> 10.0\VC\INCLUDE\xlocale(323) >> : warning C4530: C++ exception handler used, but unwind semantics are not >> enabled. Specify /EHsc >> """ >> >> Stefan >> > > > Thanks, > > Christoph > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From cgohlke at uci.edu Sun Aug 26 17:54:53 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 26 Aug 2012 08:54:53 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <5039EF53.50806@behnel.de> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <5038D85A.6090505@behnel.de> <5039D85C.1090600@uci.edu> <5039EE2C.3080100@behnel.de> <5039EF53.50806@behnel.de> Message-ID: <503A46CD.9010109@uci.edu> On 8/26/2012 2:41 AM, Stefan Behnel wrote: > Stefan Behnel, 26.08.2012 11:36: >> Christoph Gohlke, 26.08.2012 10:03: >>> On 8/25/2012 6:51 AM, Stefan Behnel wrote: >>>> Christoph Gohlke, 25.08.2012 04:07: >>>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>>>> Christoph Gohlke, 24.08.2012 07:20: >>>>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>>>> during >>>>>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>>>>>> Windows executive can not identify in which specific module it >>>>>>> crashes, and >>>>>>> neither enabling faulthandler nor building with debug symbols gives any >>>>>>> useful information. Can anyone reproduce this? It seems compiler specific >>>>>>> since Python 3.3, which is using msvc10, does not crash. >>>>>> >>>>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>>>> this sorted out, but it's almost impossible to debug something like this >>>>>> from a distance. >>>>> >>>>> Maybe the following simple example is related. It fails (not crash) when >>>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 >>>>> and 64 bit): >>>>> >>>>> ``` >>>>> from cython.view cimport array as cvarray >>>>> import numpy as np >>>>> >>>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>>>> >>>>> cdef int[:, :, ::1] a = narr >>>>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>>>> >>>>> print narr[2:8:2, -4:1:-1, 1:3].shape >>>>> print b.shape[0], b.shape[1], b.shape[2] >>>>> ``` >>>>> >>>>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >>>> >>>> I'll leave that for the others to comment. >>> >>> This failure and the test crash disappear when auto-inlining is disabled, >>> e.g. using the `/Ob1` compiler switch >>> >> >> Thanks for figuring that out. >> >> >>> Is there a way to add a C pre-processor `#pragma` to a Cython generated C >>> file? >> >> I don't think so, not in arbitrary places. >> >> >>> I tried `cdef void emit_pragma '#pragma auto_inline(off) //' ()` but >>> the pragma ends up inside the Python module init function, which fails to >>> compile with `error C2156: pragma must be outside function`. >> >> As a quick fix, we could wrap (or just start) our utility code section with >> that pragma. It starts after all user code. According to the docs, the >> pragma only disables automatic inlining in the specific range, not >> explicitly requested inlining. >> >> http://msdn.microsoft.com/en-us/library/ah08w5c3%28v=vs.71%29.aspx >> >> We usually know what we consider worth inlining and what doesn't need to >> be. If we can find out what exact functions trigger this, we can start >> being more specific about where to enable the pragma, but I think it's ok >> to enable it on all utility code for now. >> >> Could you figure out a suitable preprocessor guard that only enables it for >> the affected compilers? How about `#if defined(_WIN64) && (_MSC_VER == 1500)` > > A patch would essentially look like this: > > diff -r 18fed0dec20e Cython/Compiler/Code.py > --- a/Cython/Compiler/Code.py Sun Aug 26 00:54:01 2012 +0200 > +++ b/Cython/Compiler/Code.py Sun Aug 26 11:39:29 2012 +0200 > @@ -893,6 +893,8 @@ > code.write('\n#line 1 "cython_utility"\n') > code.putln("") > code.putln("/* Runtime support code */") > + code.putln("#pragma auto_inline(off)") > + code.putln("") > > def finalize_main_c_code(self): > self.close_global_decls() > > Stefan > Thanks. I just tried this. The `#pragma auto_inline(off)` line appears after the MemoryView code and has no effect on that part. Christoph From cgohlke at uci.edu Sun Aug 26 18:32:44 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 26 Aug 2012 09:32:44 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> Message-ID: <503A4FAC.7080203@uci.edu> On 8/26/2012 4:08 AM, mark florisson wrote: > On 25 August 2012 03:07, Christoph Gohlke wrote: >> Hi, >> >> >> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>> >>> Hi, >>> >>> thanks for testing! >>> >>> Christoph Gohlke, 24.08.2012 07:20: >>>> >>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>> >>>> 32 bit Python 2.7 works well, only 4 test failures. >>> >>> >>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>> build environment? >> >> >> OpenMP is available on my system, and parallel.pyd is linked to the openmp >> library. The prange tests fail only sometimes. On my system, the prange >> index is sometimes left at the start (zero) of the range, while the tests >> expect the index to be left at the stop of the range. According to the >> Cython prange enhancements webpage "the iterations of the loop body can be >> executed in any order" . Where >> does that leave the loop index? >> > > I should be the value from the last iteration, but in my experience > many compilers have buggy OpenMP implementations. I think your > compiler doesn't correctly support the lastprivate clause in all > situations. For instance, test_prange fails (which doesn't have any > break/return/exceptions), which simply has a lastprivate clause and a > schedule clause set to 'dynamic'. The test without the schedule clause > works fine (or maybe it's just luck). Or maybe it doesn't support > multiple lastprivate() clauses? I'm not entirely sure... It also seems > the thread limit on your system is 1. Thanks. Good to know. Regarding the thread limit of 1: according to , if omp_get_num_threads is "called from a serial portion of a program, or from a nested parallel region that is serialized, this function returns 1." This seems to be the case here (looking at the generated C code for the test_num_threads function). In the example at they add a `#pragma omp master` directive before calling omp_get_num_threads( ). Christoph > > In any case, the generated code for these tests looks correct to me, > but we've had similar problems before with different compilers... > >> >>> >>> The other one is the new "initial_file_path" test that fails with this >>> linker error: >>> >>> """ >>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>> /LIBPATH:X:\Python27\libs >>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>> >>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>> >>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>> >>> LINK : error LNK2001: unresolved external symbol init__init__ >>> """ >>> >>> Maybe the Windows build of distutils is broken here - it seems to assume >>> the wrong module name for the package module. >> >> >> I think this is an issue with the test. The extension does compile and link >> outside of the tests. >> >> >> >>> >>> I guess compiling package modules is just an overall badly supported >>> feature in CPython... >>> >>> >>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>> during >>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>>> Windows executive can not identify in which specific module it crashes, >>>> and >>>> neither enabling faulthandler nor building with debug symbols gives any >>>> useful information. Can anyone reproduce this? It seems compiler specific >>>> since Python 3.3, which is using msvc10, does not crash. >>> >>> >>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>> this sorted out, but it's almost impossible to debug something like this >>> from a distance. >> >> >> >> Maybe the following simple example is related. It fails (not crash) when >> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 >> and 64 bit): >> >> ``` >> from cython.view cimport array as cvarray >> import numpy as np >> >> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >> >> cdef int[:, :, ::1] a = narr >> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >> >> print narr[2:8:2, -4:1:-1, 1:3].shape >> print b.shape[0], b.shape[1], b.shape[2] >> ``` >> >> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >> >> >> >>> >>> >>>> When disabling >>>> test_slice_assignment, runtests.py completes with many failures. >>>> >>>> The results of `runtests.py -v -v` are at >>>> . >>> >>> >>> The 64bit output looks so broken that I wonder what went wrong here. I >>> mean, most of the problems look like this: >>> >>> """ >>> Expected: >>> Traceback (most recent call last): >>> TypeError: m() takes at most 2 positional arguments (3 given) >>> Got: >>> Traceback (most recent call last): >>> TypeError: m() takes at most %Id positional argument%s (%Id given) >>> """ >>> >>> I have no idea how that can happen. >>> >>> I can see two other problems, one is the linker warning about the module >>> init function in Py3: >>> >>> """ >>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>> multiple times; using first specification >>> """ >> >> >> This is "normal". >> >> >> >>> >>> The other one is about mixing Py_ssize_t and int (and some other) types >>> all >>> over the place: >>> >>> """ >>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >>> 'int', possible loss of data >>> """ >>> >>> Some of them look like we'd need an explicit cast in the C code somewhere, >>> others might hint at lax type usage in tests. >>> >>> There's also this: >>> >>> """ >>> C:\Program Files (x86)\Microsoft Visual Studio >>> 10.0\VC\INCLUDE\xlocale(323) >>> : warning C4530: C++ exception handler used, but unwind semantics are not >>> enabled. Specify /EHsc >>> """ >>> >>> Stefan >>> >> >> >> Thanks, >> >> Christoph >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > From stefan_ml at behnel.de Sun Aug 26 18:47:53 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 26 Aug 2012 18:47:53 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503A46CD.9010109@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <5038D85A.6090505@behnel.de> <5039D85C.1090600@uci.edu> <5039EE2C.3080100@behnel.de> <5039EF53.50806@behnel.de> <503A46CD.9010109@uci.edu> Message-ID: <503A5339.6050600@behnel.de> Christoph Gohlke, 26.08.2012 17:54: > On 8/26/2012 2:41 AM, Stefan Behnel wrote: >> A patch would essentially look like this: >> >> diff -r 18fed0dec20e Cython/Compiler/Code.py >> --- a/Cython/Compiler/Code.py Sun Aug 26 00:54:01 2012 +0200 >> +++ b/Cython/Compiler/Code.py Sun Aug 26 11:39:29 2012 +0200 >> @@ -893,6 +893,8 @@ >> code.write('\n#line 1 "cython_utility"\n') >> code.putln("") >> code.putln("/* Runtime support code */") >> + code.putln("#pragma auto_inline(off)") >> + code.putln("") >> >> def finalize_main_c_code(self): >> self.close_global_decls() > > Thanks. I just tried this. The `#pragma auto_inline(off)` line appears > after the MemoryView code and has no effect on that part. Argh. That's too bad - the memory view code is actually compiled from Cython code and gets injected at the tree level, not by the code generator. I would like to avoid an impact on user code by this pragma, so I think we'll have to find a way to inject the pragma into the code stream somehow... Stefan From markflorisson88 at gmail.com Sun Aug 26 20:08:38 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 26 Aug 2012 19:08:38 +0100 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503A4FAC.7080203@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503A4FAC.7080203@uci.edu> Message-ID: On 26 August 2012 17:32, Christoph Gohlke wrote: > On 8/26/2012 4:08 AM, mark florisson wrote: >> >> On 25 August 2012 03:07, Christoph Gohlke wrote: >>> >>> Hi, >>> >>> >>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>> >>>> >>>> Hi, >>>> >>>> thanks for testing! >>>> >>>> Christoph Gohlke, 24.08.2012 07:20: >>>>> >>>>> >>>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>>> >>>>> 32 bit Python 2.7 works well, only 4 test failures. >>>> >>>> >>>> >>>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>>> build environment? >>> >>> >>> >>> OpenMP is available on my system, and parallel.pyd is linked to the >>> openmp >>> library. The prange tests fail only sometimes. On my system, the prange >>> index is sometimes left at the start (zero) of the range, while the tests >>> expect the index to be left at the stop of the range. According to the >>> Cython prange enhancements webpage "the iterations of the loop body can >>> be >>> executed in any order" . >>> Where >>> does that leave the loop index? >>> >> >> I should be the value from the last iteration, but in my experience >> many compilers have buggy OpenMP implementations. I think your >> compiler doesn't correctly support the lastprivate clause in all >> situations. For instance, test_prange fails (which doesn't have any >> break/return/exceptions), which simply has a lastprivate clause and a >> schedule clause set to 'dynamic'. The test without the schedule clause >> works fine (or maybe it's just luck). Or maybe it doesn't support >> multiple lastprivate() clauses? I'm not entirely sure... It also seems >> the thread limit on your system is 1. > > > Thanks. Good to know. > > Regarding the thread limit of 1: according to > , if > omp_get_num_threads is "called from a serial portion of a program, or from a > nested parallel region that is serialized, this function returns 1." This > seems to be the case here (looking at the generated C code for the > test_num_threads function). I don't think so, all calls are made from within parallel sections in the test code, and the corresponding generated code. But OpenMP implementations are allowed to give you less threads if the implementation doesn't support more, or if the OS doesn't allow more. In any case, that test is kind of a non-problem. > In the example at > they > add a `#pragma omp master` directive before calling omp_get_num_threads( ). That's because they only need one assignment and call. > Christoph > > > >> >> In any case, the generated code for these tests looks correct to me, >> but we've had similar problems before with different compilers... >> >>> >>>> >>>> The other one is the new "initial_file_path" test that fails with this >>>> linker error: >>>> >>>> """ >>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>>> /LIBPATH:X:\Python27\libs >>>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>>> >>>> >>>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>>> >>>> >>>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>>> >>>> LINK : error LNK2001: unresolved external symbol init__init__ >>>> """ >>>> >>>> Maybe the Windows build of distutils is broken here - it seems to assume >>>> the wrong module name for the package module. >>> >>> >>> >>> I think this is an issue with the test. The extension does compile and >>> link >>> outside of the tests. >>> >>> >>> >>>> >>>> I guess compiling package modules is just an overall badly supported >>>> feature in CPython... >>>> >>>> >>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>> during >>>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. >>>>> The >>>>> Windows executive can not identify in which specific module it crashes, >>>>> and >>>>> neither enabling faulthandler nor building with debug symbols gives any >>>>> useful information. Can anyone reproduce this? It seems compiler >>>>> specific >>>>> since Python 3.3, which is using msvc10, does not crash. >>>> >>>> >>>> >>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>> this sorted out, but it's almost impossible to debug something like this >>>> from a distance. >>> >>> >>> >>> >>> Maybe the following simple example is related. It fails (not crash) when >>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 >>> (32 >>> and 64 bit): >>> >>> ``` >>> from cython.view cimport array as cvarray >>> import numpy as np >>> >>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>> >>> cdef int[:, :, ::1] a = narr >>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>> >>> print narr[2:8:2, -4:1:-1, 1:3].shape >>> print b.shape[0], b.shape[1], b.shape[2] >>> ``` >>> >>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >>> >>> >>> >>>> >>>> >>>>> When disabling >>>>> test_slice_assignment, runtests.py completes with many failures. >>>>> >>>>> The results of `runtests.py -v -v` are at >>>>> . >>>> >>>> >>>> >>>> The 64bit output looks so broken that I wonder what went wrong here. I >>>> mean, most of the problems look like this: >>>> >>>> """ >>>> Expected: >>>> Traceback (most recent call last): >>>> TypeError: m() takes at most 2 positional arguments (3 given) >>>> Got: >>>> Traceback (most recent call last): >>>> TypeError: m() takes at most %Id positional argument%s (%Id given) >>>> """ >>>> >>>> I have no idea how that can happen. >>>> >>>> I can see two other problems, one is the linker warning about the module >>>> init function in Py3: >>>> >>>> """ >>>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>>> multiple times; using first specification >>>> """ >>> >>> >>> >>> This is "normal". >>> >>> >>> >>>> >>>> The other one is about mixing Py_ssize_t and int (and some other) types >>>> all >>>> over the place: >>>> >>>> """ >>>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >>>> 'int', possible loss of data >>>> """ >>>> >>>> Some of them look like we'd need an explicit cast in the C code >>>> somewhere, >>>> others might hint at lax type usage in tests. >>>> >>>> There's also this: >>>> >>>> """ >>>> C:\Program Files (x86)\Microsoft Visual Studio >>>> 10.0\VC\INCLUDE\xlocale(323) >>>> : warning C4530: C++ exception handler used, but unwind semantics are >>>> not >>>> enabled. Specify /EHsc >>>> """ >>>> >>>> Stefan >>>> >>> >>> >>> Thanks, >>> >>> Christoph >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> >> >> > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From cgohlke at uci.edu Sun Aug 26 21:06:30 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 26 Aug 2012 12:06:30 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503A4FAC.7080203@uci.edu> Message-ID: <503A73B6.3010006@uci.edu> On 8/26/2012 11:08 AM, mark florisson wrote: > On 26 August 2012 17:32, Christoph Gohlke wrote: >> On 8/26/2012 4:08 AM, mark florisson wrote: >>> >>> On 25 August 2012 03:07, Christoph Gohlke wrote: >>>> >>>> Hi, >>>> >>>> >>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>>> >>>>> >>>>> Hi, >>>>> >>>>> thanks for testing! >>>>> >>>>> Christoph Gohlke, 24.08.2012 07:20: >>>>>> >>>>>> >>>>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>>>> >>>>>> 32 bit Python 2.7 works well, only 4 test failures. >>>>> >>>>> >>>>> >>>>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>>>> build environment? >>>> >>>> >>>> >>>> OpenMP is available on my system, and parallel.pyd is linked to the >>>> openmp >>>> library. The prange tests fail only sometimes. On my system, the prange >>>> index is sometimes left at the start (zero) of the range, while the tests >>>> expect the index to be left at the stop of the range. According to the >>>> Cython prange enhancements webpage "the iterations of the loop body can >>>> be >>>> executed in any order" . >>>> Where >>>> does that leave the loop index? >>>> >>> >>> I should be the value from the last iteration, but in my experience >>> many compilers have buggy OpenMP implementations. I think your >>> compiler doesn't correctly support the lastprivate clause in all >>> situations. For instance, test_prange fails (which doesn't have any >>> break/return/exceptions), which simply has a lastprivate clause and a >>> schedule clause set to 'dynamic'. The test without the schedule clause >>> works fine (or maybe it's just luck). Or maybe it doesn't support >>> multiple lastprivate() clauses? I'm not entirely sure... It also seems >>> the thread limit on your system is 1. >> >> >> Thanks. Good to know. >> >> Regarding the thread limit of 1: according to >> , if >> omp_get_num_threads is "called from a serial portion of a program, or from a >> nested parallel region that is serialized, this function returns 1." This >> seems to be the case here (looking at the generated C code for the >> test_num_threads function). > > I don't think so, all calls are made from within parallel sections in > the test code, and the corresponding generated code. But OpenMP > implementations are allowed to give you less threads if the > implementation doesn't support more, or if the OS doesn't allow more. > In any case, that test is kind of a non-problem. > >> In the example at >> they >> add a `#pragma omp master` directive before calling omp_get_num_threads( ). > > That's because they only need one assignment and call. You are right of course. Sorry. I verified that with a simpler example. It seems that runtests.py does not add the `/openmp` flag to the compiler arguments when compiling C++ code. If I change line 119 in runtests.py to return `'/openmp', ''` instead of None the test_num_threads failure disappears. But the cpp\parallel\prange tests start to fail too. Christoph >> >> >> >>> >>> In any case, the generated code for these tests looks correct to me, >>> but we've had similar problems before with different compilers... >>> >>>> >>>>> >>>>> The other one is the new "initial_file_path" test that fails with this >>>>> linker error: >>>>> >>>>> """ >>>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>>>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>>>> /LIBPATH:X:\Python27\libs >>>>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>>>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>>>> >>>>> >>>>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>>>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>>>> >>>>> >>>>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>>>> >>>>> LINK : error LNK2001: unresolved external symbol init__init__ >>>>> """ >>>>> >>>>> Maybe the Windows build of distutils is broken here - it seems to assume >>>>> the wrong module name for the package module. >>>> >>>> >>>> >>>> I think this is an issue with the test. The extension does compile and >>>> link >>>> outside of the tests. >>>> >>>> >>>> >>>>> >>>>> I guess compiling package modules is just an overall badly supported >>>>> feature in CPython... >>>>> >>>>> >>>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>>> during >>>>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. >>>>>> The >>>>>> Windows executive can not identify in which specific module it crashes, >>>>>> and >>>>>> neither enabling faulthandler nor building with debug symbols gives any >>>>>> useful information. Can anyone reproduce this? It seems compiler >>>>>> specific >>>>>> since Python 3.3, which is using msvc10, does not crash. >>>>> >>>>> >>>>> >>>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>>> this sorted out, but it's almost impossible to debug something like this >>>>> from a distance. >>>> >>>> >>>> >>>> >>>> Maybe the following simple example is related. It fails (not crash) when >>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 >>>> (32 >>>> and 64 bit): >>>> >>>> ``` >>>> from cython.view cimport array as cvarray >>>> import numpy as np >>>> >>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>>> >>>> cdef int[:, :, ::1] a = narr >>>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>>> >>>> print narr[2:8:2, -4:1:-1, 1:3].shape >>>> print b.shape[0], b.shape[1], b.shape[2] >>>> ``` >>>> >>>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >>>> >>>> >>>> >>>>> >>>>> >>>>>> When disabling >>>>>> test_slice_assignment, runtests.py completes with many failures. >>>>>> >>>>>> The results of `runtests.py -v -v` are at >>>>>> . >>>>> >>>>> >>>>> >>>>> The 64bit output looks so broken that I wonder what went wrong here. I >>>>> mean, most of the problems look like this: >>>>> >>>>> """ >>>>> Expected: >>>>> Traceback (most recent call last): >>>>> TypeError: m() takes at most 2 positional arguments (3 given) >>>>> Got: >>>>> Traceback (most recent call last): >>>>> TypeError: m() takes at most %Id positional argument%s (%Id given) >>>>> """ >>>>> >>>>> I have no idea how that can happen. >>>>> >>>>> I can see two other problems, one is the linker warning about the module >>>>> init function in Py3: >>>>> >>>>> """ >>>>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>>>> multiple times; using first specification >>>>> """ >>>> >>>> >>>> >>>> This is "normal". >>>> >>>> >>>> >>>>> >>>>> The other one is about mixing Py_ssize_t and int (and some other) types >>>>> all >>>>> over the place: >>>>> >>>>> """ >>>>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >>>>> 'int', possible loss of data >>>>> """ >>>>> >>>>> Some of them look like we'd need an explicit cast in the C code >>>>> somewhere, >>>>> others might hint at lax type usage in tests. >>>>> >>>>> There's also this: >>>>> >>>>> """ >>>>> C:\Program Files (x86)\Microsoft Visual Studio >>>>> 10.0\VC\INCLUDE\xlocale(323) >>>>> : warning C4530: C++ exception handler used, but unwind semantics are >>>>> not >>>>> enabled. Specify /EHsc >>>>> """ >>>>> >>>>> Stefan >>>>> >>>> >>>> >>>> Thanks, >>>> >>>> Christoph >>>> >>>> _______________________________________________ >>>> cython-devel mailing list >>>> cython-devel at python.org >>>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> >>> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > From cgohlke at uci.edu Sun Aug 26 23:09:37 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 26 Aug 2012 14:09:37 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> Message-ID: <503A9091.5040308@uci.edu> On 8/26/2012 4:08 AM, mark florisson wrote: > On 25 August 2012 03:07, Christoph Gohlke wrote: >> Hi, >> >> >> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>> >>> Hi, >>> >>> thanks for testing! >>> >>> Christoph Gohlke, 24.08.2012 07:20: >>>> >>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>> >>>> 32 bit Python 2.7 works well, only 4 test failures. >>> >>> >>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>> build environment? >> >> >> OpenMP is available on my system, and parallel.pyd is linked to the openmp >> library. The prange tests fail only sometimes. On my system, the prange >> index is sometimes left at the start (zero) of the range, while the tests >> expect the index to be left at the stop of the range. According to the >> Cython prange enhancements webpage "the iterations of the loop body can be >> executed in any order" . Where >> does that leave the loop index? >> > > I should be the value from the last iteration, but in my experience > many compilers have buggy OpenMP implementations. I think your > compiler doesn't correctly support the lastprivate clause in all > situations. For instance, test_prange fails (which doesn't have any > break/return/exceptions), which simply has a lastprivate clause and a > schedule clause set to 'dynamic'. The test without the schedule clause > works fine (or maybe it's just luck). Or maybe it doesn't support > multiple lastprivate() clauses? I'm not entirely sure... It also seems > the thread limit on your system is 1. > > In any case, the generated code for these tests looks correct to me, > but we've had similar problems before with different compilers... A minimal example that fails for me is: def test_parallel(): cdef int i = 0, s = 0 with nogil, cython.parallel.parallel(): for i in prange(10): s += i return i The returned value is often 0, otherwise 9 as expected. In the generated C code I see #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) If I change this to #pragma omp parallel for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) the function always returns the expected value. Does that make sense? Thanks for your help and patience. Christoph > >> >>> >>> The other one is the new "initial_file_path" test that fails with this >>> linker error: >>> >>> """ >>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>> /LIBPATH:X:\Python27\libs >>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>> >>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>> >>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>> >>> LINK : error LNK2001: unresolved external symbol init__init__ >>> """ >>> >>> Maybe the Windows build of distutils is broken here - it seems to assume >>> the wrong module name for the package module. >> >> >> I think this is an issue with the test. The extension does compile and link >> outside of the tests. >> >> >> >>> >>> I guess compiling package modules is just an overall badly supported >>> feature in CPython... >>> >>> >>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>> during >>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>>> Windows executive can not identify in which specific module it crashes, >>>> and >>>> neither enabling faulthandler nor building with debug symbols gives any >>>> useful information. Can anyone reproduce this? It seems compiler specific >>>> since Python 3.3, which is using msvc10, does not crash. >>> >>> >>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>> this sorted out, but it's almost impossible to debug something like this >>> from a distance. >> >> >> >> Maybe the following simple example is related. It fails (not crash) when >> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 >> and 64 bit): >> >> ``` >> from cython.view cimport array as cvarray >> import numpy as np >> >> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >> >> cdef int[:, :, ::1] a = narr >> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >> >> print narr[2:8:2, -4:1:-1, 1:3].shape >> print b.shape[0], b.shape[1], b.shape[2] >> ``` >> >> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >> >> >> >>> >>> >>>> When disabling >>>> test_slice_assignment, runtests.py completes with many failures. >>>> >>>> The results of `runtests.py -v -v` are at >>>> . >>> >>> >>> The 64bit output looks so broken that I wonder what went wrong here. I >>> mean, most of the problems look like this: >>> >>> """ >>> Expected: >>> Traceback (most recent call last): >>> TypeError: m() takes at most 2 positional arguments (3 given) >>> Got: >>> Traceback (most recent call last): >>> TypeError: m() takes at most %Id positional argument%s (%Id given) >>> """ >>> >>> I have no idea how that can happen. >>> >>> I can see two other problems, one is the linker warning about the module >>> init function in Py3: >>> >>> """ >>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>> multiple times; using first specification >>> """ >> >> >> This is "normal". >> >> >> >>> >>> The other one is about mixing Py_ssize_t and int (and some other) types >>> all >>> over the place: >>> >>> """ >>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >>> 'int', possible loss of data >>> """ >>> >>> Some of them look like we'd need an explicit cast in the C code somewhere, >>> others might hint at lax type usage in tests. >>> >>> There's also this: >>> >>> """ >>> C:\Program Files (x86)\Microsoft Visual Studio >>> 10.0\VC\INCLUDE\xlocale(323) >>> : warning C4530: C++ exception handler used, but unwind semantics are not >>> enabled. Specify /EHsc >>> """ >>> >>> Stefan >>> >> >> >> Thanks, >> >> Christoph >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > From cgohlke at uci.edu Sun Aug 26 23:25:16 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 26 Aug 2012 14:25:16 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503A9091.5040308@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503A9091.5040308@uci.edu> Message-ID: <503A943C.7000602@uci.edu> On 8/26/2012 2:09 PM, Christoph Gohlke wrote: > On 8/26/2012 4:08 AM, mark florisson wrote: >> On 25 August 2012 03:07, Christoph Gohlke wrote: >>> Hi, >>> >>> >>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>> >>>> Hi, >>>> >>>> thanks for testing! >>>> >>>> Christoph Gohlke, 24.08.2012 07:20: >>>>> >>>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>>> >>>>> 32 bit Python 2.7 works well, only 4 test failures. >>>> >>>> >>>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>>> build environment? >>> >>> >>> OpenMP is available on my system, and parallel.pyd is linked to the >>> openmp >>> library. The prange tests fail only sometimes. On my system, the prange >>> index is sometimes left at the start (zero) of the range, while the >>> tests >>> expect the index to be left at the stop of the range. According to the >>> Cython prange enhancements webpage "the iterations of the loop body >>> can be >>> executed in any order" . >>> Where >>> does that leave the loop index? >>> >> >> I should be the value from the last iteration, but in my experience >> many compilers have buggy OpenMP implementations. I think your >> compiler doesn't correctly support the lastprivate clause in all >> situations. For instance, test_prange fails (which doesn't have any >> break/return/exceptions), which simply has a lastprivate clause and a >> schedule clause set to 'dynamic'. The test without the schedule clause >> works fine (or maybe it's just luck). Or maybe it doesn't support >> multiple lastprivate() clauses? I'm not entirely sure... It also seems >> the thread limit on your system is 1. >> >> In any case, the generated code for these tests looks correct to me, >> but we've had similar problems before with different compilers... > > A minimal example that fails for me is: > > def test_parallel(): > cdef int i = 0, s = 0 > with nogil, cython.parallel.parallel(): > for i in prange(10): > s += i > return i > > The returned value is often 0, otherwise 9 as expected. > > In the generated C code I see > > #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) > > If I change this to > > #pragma omp parallel for firstprivate(__pyx_v_i) > lastprivate(__pyx_v_i) > > the function always returns the expected value. Does that make sense? Well, apparently it doesn't make sense because the value of s is not correct. Christoph > > Thanks for your help and patience. > > Christoph > > >> >>> >>>> >>>> The other one is the new "initial_file_path" test that fails with this >>>> linker error: >>>> >>>> """ >>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>>> /LIBPATH:X:\Python27\libs >>>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>>> >>>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>>> >>>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>>> >>>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>>> >>>> >>>> LINK : error LNK2001: unresolved external symbol init__init__ >>>> """ >>>> >>>> Maybe the Windows build of distutils is broken here - it seems to >>>> assume >>>> the wrong module name for the package module. >>> >>> >>> I think this is an issue with the test. The extension does compile >>> and link >>> outside of the tests. >>> >>> >>> >>>> >>>> I guess compiling package modules is just an overall badly supported >>>> feature in CPython... >>>> >>>> >>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>> during >>>>> `test_slice_assignment (memslice.__test__)`. I tested two >>>>> computers. The >>>>> Windows executive can not identify in which specific module it >>>>> crashes, >>>>> and >>>>> neither enabling faulthandler nor building with debug symbols gives >>>>> any >>>>> useful information. Can anyone reproduce this? It seems compiler >>>>> specific >>>>> since Python 3.3, which is using msvc10, does not crash. >>>> >>>> >>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>> this sorted out, but it's almost impossible to debug something like >>>> this >>>> from a distance. >>> >>> >>> >>> Maybe the following simple example is related. It fails (not crash) when >>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and >>> msvc10 (32 >>> and 64 bit): >>> >>> ``` >>> from cython.view cimport array as cvarray >>> import numpy as np >>> >>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>> >>> cdef int[:, :, ::1] a = narr >>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>> >>> print narr[2:8:2, -4:1:-1, 1:3].shape >>> print b.shape[0], b.shape[1], b.shape[2] >>> ``` >>> >>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, >>> 9, 2) >>> >>> >>> >>>> >>>> >>>>> When disabling >>>>> test_slice_assignment, runtests.py completes with many failures. >>>>> >>>>> The results of `runtests.py -v -v` are at >>>>> . >>>> >>>> >>>> The 64bit output looks so broken that I wonder what went wrong here. I >>>> mean, most of the problems look like this: >>>> >>>> """ >>>> Expected: >>>> Traceback (most recent call last): >>>> TypeError: m() takes at most 2 positional arguments (3 given) >>>> Got: >>>> Traceback (most recent call last): >>>> TypeError: m() takes at most %Id positional argument%s (%Id >>>> given) >>>> """ >>>> >>>> I have no idea how that can happen. >>>> >>>> I can see two other problems, one is the linker warning about the >>>> module >>>> init function in Py3: >>>> >>>> """ >>>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>>> multiple times; using first specification >>>> """ >>> >>> >>> This is "normal". >>> >>> >>> >>>> >>>> The other one is about mixing Py_ssize_t and int (and some other) types >>>> all >>>> over the place: >>>> >>>> """ >>>> bufaccess.c(3033) : warning C4244: '=' : conversion from >>>> 'Py_ssize_t' to >>>> 'int', possible loss of data >>>> """ >>>> >>>> Some of them look like we'd need an explicit cast in the C code >>>> somewhere, >>>> others might hint at lax type usage in tests. >>>> >>>> There's also this: >>>> >>>> """ >>>> C:\Program Files (x86)\Microsoft Visual Studio >>>> 10.0\VC\INCLUDE\xlocale(323) >>>> : warning C4530: C++ exception handler used, but unwind semantics >>>> are not >>>> enabled. Specify /EHsc >>>> """ >>>> >>>> Stefan >>>> >>> >>> >>> Thanks, >>> >>> Christoph >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> >> >> > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > From markflorisson88 at gmail.com Sun Aug 26 23:45:09 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 26 Aug 2012 22:45:09 +0100 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503A943C.7000602@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503A9091.5040308@uci.edu> <503A943C.7000602@uci.edu> Message-ID: On 26 August 2012 22:25, Christoph Gohlke wrote: > On 8/26/2012 2:09 PM, Christoph Gohlke wrote: >> >> On 8/26/2012 4:08 AM, mark florisson wrote: >>> >>> On 25 August 2012 03:07, Christoph Gohlke wrote: >>>> >>>> Hi, >>>> >>>> >>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>>> >>>>> >>>>> Hi, >>>>> >>>>> thanks for testing! >>>>> >>>>> Christoph Gohlke, 24.08.2012 07:20: >>>>>> >>>>>> >>>>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>>>> >>>>>> 32 bit Python 2.7 works well, only 4 test failures. >>>>> >>>>> >>>>> >>>>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>>>> build environment? >>>> >>>> >>>> >>>> OpenMP is available on my system, and parallel.pyd is linked to the >>>> openmp >>>> library. The prange tests fail only sometimes. On my system, the prange >>>> index is sometimes left at the start (zero) of the range, while the >>>> tests >>>> expect the index to be left at the stop of the range. According to the >>>> Cython prange enhancements webpage "the iterations of the loop body >>>> can be >>>> executed in any order" . >>>> Where >>>> does that leave the loop index? >>>> >>> >>> I should be the value from the last iteration, but in my experience >>> many compilers have buggy OpenMP implementations. I think your >>> compiler doesn't correctly support the lastprivate clause in all >>> situations. For instance, test_prange fails (which doesn't have any >>> break/return/exceptions), which simply has a lastprivate clause and a >>> schedule clause set to 'dynamic'. The test without the schedule clause >>> works fine (or maybe it's just luck). Or maybe it doesn't support >>> multiple lastprivate() clauses? I'm not entirely sure... It also seems >>> the thread limit on your system is 1. >>> >>> In any case, the generated code for these tests looks correct to me, >>> but we've had similar problems before with different compilers... >> >> >> A minimal example that fails for me is: >> >> def test_parallel(): >> cdef int i = 0, s = 0 >> with nogil, cython.parallel.parallel(): >> for i in prange(10): >> s += i >> return i >> >> The returned value is often 0, otherwise 9 as expected. >> >> In the generated C code I see >> >> #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) >> >> If I change this to >> >> #pragma omp parallel for firstprivate(__pyx_v_i) >> lastprivate(__pyx_v_i) >> >> the function always returns the expected value. Does that make sense? > > > > Well, apparently it doesn't make sense because the value of s is not > correct. > > Christoph > That will use nested parallelism. > >> >> Thanks for your help and patience. >> >> Christoph >> >> >>> >>>> >>>>> >>>>> The other one is the new "initial_file_path" test that fails with this >>>>> linker error: >>>>> >>>>> """ >>>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>>>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>>>> /LIBPATH:X:\Python27\libs >>>>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>>>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>>>> >>>>> >>>>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>>>> >>>>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>>>> >>>>> >>>>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>>>> >>>>> >>>>> LINK : error LNK2001: unresolved external symbol init__init__ >>>>> """ >>>>> >>>>> Maybe the Windows build of distutils is broken here - it seems to >>>>> assume >>>>> the wrong module name for the package module. >>>> >>>> >>>> >>>> I think this is an issue with the test. The extension does compile >>>> and link >>>> outside of the tests. >>>> >>>> >>>> >>>>> >>>>> I guess compiling package modules is just an overall badly supported >>>>> feature in CPython... >>>>> >>>>> >>>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>>> during >>>>>> `test_slice_assignment (memslice.__test__)`. I tested two >>>>>> computers. The >>>>>> Windows executive can not identify in which specific module it >>>>>> crashes, >>>>>> and >>>>>> neither enabling faulthandler nor building with debug symbols gives >>>>>> any >>>>>> useful information. Can anyone reproduce this? It seems compiler >>>>>> specific >>>>>> since Python 3.3, which is using msvc10, does not crash. >>>>> >>>>> >>>>> >>>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>>> this sorted out, but it's almost impossible to debug something like >>>>> this >>>>> from a distance. >>>> >>>> >>>> >>>> >>>> Maybe the following simple example is related. It fails (not crash) when >>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and >>>> msvc10 (32 >>>> and 64 bit): >>>> >>>> ``` >>>> from cython.view cimport array as cvarray >>>> import numpy as np >>>> >>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>>> >>>> cdef int[:, :, ::1] a = narr >>>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>>> >>>> print narr[2:8:2, -4:1:-1, 1:3].shape >>>> print b.shape[0], b.shape[1], b.shape[2] >>>> ``` >>>> >>>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, >>>> 9, 2) >>>> >>>> >>>> >>>>> >>>>> >>>>>> When disabling >>>>>> test_slice_assignment, runtests.py completes with many failures. >>>>>> >>>>>> The results of `runtests.py -v -v` are at >>>>>> . >>>>> >>>>> >>>>> >>>>> The 64bit output looks so broken that I wonder what went wrong here. I >>>>> mean, most of the problems look like this: >>>>> >>>>> """ >>>>> Expected: >>>>> Traceback (most recent call last): >>>>> TypeError: m() takes at most 2 positional arguments (3 given) >>>>> Got: >>>>> Traceback (most recent call last): >>>>> TypeError: m() takes at most %Id positional argument%s (%Id >>>>> given) >>>>> """ >>>>> >>>>> I have no idea how that can happen. >>>>> >>>>> I can see two other problems, one is the linker warning about the >>>>> module >>>>> init function in Py3: >>>>> >>>>> """ >>>>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>>>> multiple times; using first specification >>>>> """ >>>> >>>> >>>> >>>> This is "normal". >>>> >>>> >>>> >>>>> >>>>> The other one is about mixing Py_ssize_t and int (and some other) types >>>>> all >>>>> over the place: >>>>> >>>>> """ >>>>> bufaccess.c(3033) : warning C4244: '=' : conversion from >>>>> 'Py_ssize_t' to >>>>> 'int', possible loss of data >>>>> """ >>>>> >>>>> Some of them look like we'd need an explicit cast in the C code >>>>> somewhere, >>>>> others might hint at lax type usage in tests. >>>>> >>>>> There's also this: >>>>> >>>>> """ >>>>> C:\Program Files (x86)\Microsoft Visual Studio >>>>> 10.0\VC\INCLUDE\xlocale(323) >>>>> : warning C4530: C++ exception handler used, but unwind semantics >>>>> are not >>>>> enabled. Specify /EHsc >>>>> """ >>>>> >>>>> Stefan >>>>> >>>> >>>> >>>> Thanks, >>>> >>>> Christoph >>>> >>>> _______________________________________________ >>>> cython-devel mailing list >>>> cython-devel at python.org >>>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> >>> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> >> >> > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Sun Aug 26 23:45:51 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 26 Aug 2012 22:45:51 +0100 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503A9091.5040308@uci.edu> <503A943C.7000602@uci.edu> Message-ID: On 26 August 2012 22:45, mark florisson wrote: > On 26 August 2012 22:25, Christoph Gohlke wrote: >> On 8/26/2012 2:09 PM, Christoph Gohlke wrote: >>> >>> On 8/26/2012 4:08 AM, mark florisson wrote: >>>> >>>> On 25 August 2012 03:07, Christoph Gohlke wrote: >>>>> >>>>> Hi, >>>>> >>>>> >>>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> thanks for testing! >>>>>> >>>>>> Christoph Gohlke, 24.08.2012 07:20: >>>>>>> >>>>>>> >>>>>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>>>>> >>>>>>> 32 bit Python 2.7 works well, only 4 test failures. >>>>>> >>>>>> >>>>>> >>>>>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>>>>> build environment? >>>>> >>>>> >>>>> >>>>> OpenMP is available on my system, and parallel.pyd is linked to the >>>>> openmp >>>>> library. The prange tests fail only sometimes. On my system, the prange >>>>> index is sometimes left at the start (zero) of the range, while the >>>>> tests >>>>> expect the index to be left at the stop of the range. According to the >>>>> Cython prange enhancements webpage "the iterations of the loop body >>>>> can be >>>>> executed in any order" . >>>>> Where >>>>> does that leave the loop index? >>>>> >>>> >>>> I should be the value from the last iteration, but in my experience >>>> many compilers have buggy OpenMP implementations. I think your >>>> compiler doesn't correctly support the lastprivate clause in all >>>> situations. For instance, test_prange fails (which doesn't have any >>>> break/return/exceptions), which simply has a lastprivate clause and a >>>> schedule clause set to 'dynamic'. The test without the schedule clause >>>> works fine (or maybe it's just luck). Or maybe it doesn't support >>>> multiple lastprivate() clauses? I'm not entirely sure... It also seems >>>> the thread limit on your system is 1. >>>> >>>> In any case, the generated code for these tests looks correct to me, >>>> but we've had similar problems before with different compilers... >>> >>> >>> A minimal example that fails for me is: >>> >>> def test_parallel(): >>> cdef int i = 0, s = 0 >>> with nogil, cython.parallel.parallel(): >>> for i in prange(10): >>> s += i >>> return i >>> >>> The returned value is often 0, otherwise 9 as expected. >>> >>> In the generated C code I see >>> >>> #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) >>> >>> If I change this to >>> >>> #pragma omp parallel for firstprivate(__pyx_v_i) >>> lastprivate(__pyx_v_i) >>> >>> the function always returns the expected value. Does that make sense? >> >> >> >> Well, apparently it doesn't make sense because the value of s is not >> correct. >> >> Christoph >> > > That will use nested parallelism. (Which means you need 's' to be a reduction) >> >>> >>> Thanks for your help and patience. >>> >>> Christoph >>> >>> >>>> >>>>> >>>>>> >>>>>> The other one is the new "initial_file_path" test that fails with this >>>>>> linker error: >>>>>> >>>>>> """ >>>>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>>>>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>>>>> /LIBPATH:X:\Python27\libs >>>>>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>>>>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>>>>> >>>>>> >>>>>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>>>>> >>>>>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>>>>> >>>>>> >>>>>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>>>>> >>>>>> >>>>>> LINK : error LNK2001: unresolved external symbol init__init__ >>>>>> """ >>>>>> >>>>>> Maybe the Windows build of distutils is broken here - it seems to >>>>>> assume >>>>>> the wrong module name for the package module. >>>>> >>>>> >>>>> >>>>> I think this is an issue with the test. The extension does compile >>>>> and link >>>>> outside of the tests. >>>>> >>>>> >>>>> >>>>>> >>>>>> I guess compiling package modules is just an overall badly supported >>>>>> feature in CPython... >>>>>> >>>>>> >>>>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>>>> during >>>>>>> `test_slice_assignment (memslice.__test__)`. I tested two >>>>>>> computers. The >>>>>>> Windows executive can not identify in which specific module it >>>>>>> crashes, >>>>>>> and >>>>>>> neither enabling faulthandler nor building with debug symbols gives >>>>>>> any >>>>>>> useful information. Can anyone reproduce this? It seems compiler >>>>>>> specific >>>>>>> since Python 3.3, which is using msvc10, does not crash. >>>>>> >>>>>> >>>>>> >>>>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>>>> this sorted out, but it's almost impossible to debug something like >>>>>> this >>>>>> from a distance. >>>>> >>>>> >>>>> >>>>> >>>>> Maybe the following simple example is related. It fails (not crash) when >>>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and >>>>> msvc10 (32 >>>>> and 64 bit): >>>>> >>>>> ``` >>>>> from cython.view cimport array as cvarray >>>>> import numpy as np >>>>> >>>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>>>> >>>>> cdef int[:, :, ::1] a = narr >>>>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>>>> >>>>> print narr[2:8:2, -4:1:-1, 1:3].shape >>>>> print b.shape[0], b.shape[1], b.shape[2] >>>>> ``` >>>>> >>>>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, >>>>> 9, 2) >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>>>> When disabling >>>>>>> test_slice_assignment, runtests.py completes with many failures. >>>>>>> >>>>>>> The results of `runtests.py -v -v` are at >>>>>>> . >>>>>> >>>>>> >>>>>> >>>>>> The 64bit output looks so broken that I wonder what went wrong here. I >>>>>> mean, most of the problems look like this: >>>>>> >>>>>> """ >>>>>> Expected: >>>>>> Traceback (most recent call last): >>>>>> TypeError: m() takes at most 2 positional arguments (3 given) >>>>>> Got: >>>>>> Traceback (most recent call last): >>>>>> TypeError: m() takes at most %Id positional argument%s (%Id >>>>>> given) >>>>>> """ >>>>>> >>>>>> I have no idea how that can happen. >>>>>> >>>>>> I can see two other problems, one is the linker warning about the >>>>>> module >>>>>> init function in Py3: >>>>>> >>>>>> """ >>>>>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>>>>> multiple times; using first specification >>>>>> """ >>>>> >>>>> >>>>> >>>>> This is "normal". >>>>> >>>>> >>>>> >>>>>> >>>>>> The other one is about mixing Py_ssize_t and int (and some other) types >>>>>> all >>>>>> over the place: >>>>>> >>>>>> """ >>>>>> bufaccess.c(3033) : warning C4244: '=' : conversion from >>>>>> 'Py_ssize_t' to >>>>>> 'int', possible loss of data >>>>>> """ >>>>>> >>>>>> Some of them look like we'd need an explicit cast in the C code >>>>>> somewhere, >>>>>> others might hint at lax type usage in tests. >>>>>> >>>>>> There's also this: >>>>>> >>>>>> """ >>>>>> C:\Program Files (x86)\Microsoft Visual Studio >>>>>> 10.0\VC\INCLUDE\xlocale(323) >>>>>> : warning C4530: C++ exception handler used, but unwind semantics >>>>>> are not >>>>>> enabled. Specify /EHsc >>>>>> """ >>>>>> >>>>>> Stefan >>>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Christoph >>>>> >>>>> _______________________________________________ >>>>> cython-devel mailing list >>>>> cython-devel at python.org >>>>> http://mail.python.org/mailman/listinfo/cython-devel >>>> >>>> _______________________________________________ >>>> cython-devel mailing list >>>> cython-devel at python.org >>>> http://mail.python.org/mailman/listinfo/cython-devel >>>> >>>> >>>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> >>> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel From cgohlke at uci.edu Mon Aug 27 02:39:15 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Sun, 26 Aug 2012 17:39:15 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> Message-ID: <503AC1B3.90807@uci.edu> On 8/26/2012 4:08 AM, mark florisson wrote: > On 25 August 2012 03:07, Christoph Gohlke wrote: >> Hi, >> >> >> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>> >>> Hi, >>> >>> thanks for testing! >>> >>> Christoph Gohlke, 24.08.2012 07:20: >>>> >>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>> >>>> 32 bit Python 2.7 works well, only 4 test failures. >>> >>> >>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>> build environment? >> >> >> OpenMP is available on my system, and parallel.pyd is linked to the openmp >> library. The prange tests fail only sometimes. On my system, the prange >> index is sometimes left at the start (zero) of the range, while the tests >> expect the index to be left at the stop of the range. According to the >> Cython prange enhancements webpage "the iterations of the loop body can be >> executed in any order" . Where >> does that leave the loop index? >> > > I should be the value from the last iteration, but in my experience > many compilers have buggy OpenMP implementations. I think your > compiler doesn't correctly support the lastprivate clause in all > situations. For instance, test_prange fails (which doesn't have any > break/return/exceptions), which simply has a lastprivate clause and a > schedule clause set to 'dynamic'. The test without the schedule clause > works fine (or maybe it's just luck). Or maybe it doesn't support > multiple lastprivate() clauses? I'm not entirely sure... It also seems > the thread limit on your system is 1. > > In any case, the generated code for these tests looks correct to me, > but we've had similar problems before with different compilers... On my system, the following patch fixes all the lastprivate related test errors and does not have any side effects on other tests. It removes the additional initialization of the target index after `#pragma omp parallel`: diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py index 188de3d..5ba0eee 100644 --- a/Cython/Compiler/Nodes.py +++ b/Cython/Compiler/Nodes.py @@ -7627,7 +7627,7 @@ class ParallelRangeNode(ParallelStatNode): # target index uninitialized code.putln("if (%(nsteps)s > 0)" % fmt_dict) code.begin_block() # if block - code.putln("%(target)s = 0;" % fmt_dict) self.generate_loop(code, fmt_dict) code.end_block() # end if block I'm down to one remaining test failure on win32-py2.7 Christoph > >> >>> >>> The other one is the new "initial_file_path" test that fails with this >>> linker error: >>> >>> """ >>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>> /LIBPATH:X:\Python27\libs >>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>> >>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>> >>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>> >>> LINK : error LNK2001: unresolved external symbol init__init__ >>> """ >>> >>> Maybe the Windows build of distutils is broken here - it seems to assume >>> the wrong module name for the package module. >> >> >> I think this is an issue with the test. The extension does compile and link >> outside of the tests. >> >> >> >>> >>> I guess compiling package modules is just an overall badly supported >>> feature in CPython... >>> >>> >>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>> during >>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. The >>>> Windows executive can not identify in which specific module it crashes, >>>> and >>>> neither enabling faulthandler nor building with debug symbols gives any >>>> useful information. Can anyone reproduce this? It seems compiler specific >>>> since Python 3.3, which is using msvc10, does not crash. >>> >>> >>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>> this sorted out, but it's almost impossible to debug something like this >>> from a distance. >> >> >> >> Maybe the following simple example is related. It fails (not crash) when >> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 (32 >> and 64 bit): >> >> ``` >> from cython.view cimport array as cvarray >> import numpy as np >> >> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >> >> cdef int[:, :, ::1] a = narr >> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >> >> print narr[2:8:2, -4:1:-1, 1:3].shape >> print b.shape[0], b.shape[1], b.shape[2] >> ``` >> >> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >> >> >> >>> >>> >>>> When disabling >>>> test_slice_assignment, runtests.py completes with many failures. >>>> >>>> The results of `runtests.py -v -v` are at >>>> . >>> >>> >>> The 64bit output looks so broken that I wonder what went wrong here. I >>> mean, most of the problems look like this: >>> >>> """ >>> Expected: >>> Traceback (most recent call last): >>> TypeError: m() takes at most 2 positional arguments (3 given) >>> Got: >>> Traceback (most recent call last): >>> TypeError: m() takes at most %Id positional argument%s (%Id given) >>> """ >>> >>> I have no idea how that can happen. >>> >>> I can see two other problems, one is the linker warning about the module >>> init function in Py3: >>> >>> """ >>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>> multiple times; using first specification >>> """ >> >> >> This is "normal". >> >> >> >>> >>> The other one is about mixing Py_ssize_t and int (and some other) types >>> all >>> over the place: >>> >>> """ >>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >>> 'int', possible loss of data >>> """ >>> >>> Some of them look like we'd need an explicit cast in the C code somewhere, >>> others might hint at lax type usage in tests. >>> >>> There's also this: >>> >>> """ >>> C:\Program Files (x86)\Microsoft Visual Studio >>> 10.0\VC\INCLUDE\xlocale(323) >>> : warning C4530: C++ exception handler used, but unwind semantics are not >>> enabled. Specify /EHsc >>> """ >>> >>> Stefan >>> >> >> >> Thanks, >> >> Christoph >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > From lists at onerussian.com Mon Aug 27 03:38:01 2012 From: lists at onerussian.com (Yaroslav Halchenko) Date: Sun, 26 Aug 2012 21:38:01 -0400 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503672BF.5070702@behnel.de> References: <503672BF.5070702@behnel.de> Message-ID: <20120827013801.GZ5866@onerussian.com> FWIW -- this beta builds/tests ok across nearly all Debian ports but 4 where process unexpectedly terminates for no obvious reason (it built/tested just fine on my local sparc box): https://buildd.debian.org/status/package.php?p=cython&suite=experimental some time I will check WTF (memory? etc) on those intriguing ports. On Thu, 23 Aug 2012, Stefan Behnel wrote: > Hello everyone, > on behalf of the Cython project team, I'm proud to announce the release of > our third beta of Cython 0.17. This is our first and hopefully last release > candidate for 0.17, another major step forward in the development of the > language that will make life easier for a lot of users, rounds up some > rough edges of the compiler and adds (preliminary) support for CPython 3.3 > and PyPy. > Download: http://cython.org/release/Cython-0.17b3.tar.gz > Release notes: http://wiki.cython.org/ReleaseNotes-0.17 > Documentation: http://docs.cython.org/ -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik From stefan_ml at behnel.de Mon Aug 27 08:21:16 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Aug 2012 08:21:16 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <20120827013801.GZ5866@onerussian.com> References: <503672BF.5070702@behnel.de> <20120827013801.GZ5866@onerussian.com> Message-ID: <503B11DC.5030007@behnel.de> Yaroslav Halchenko, 27.08.2012 03:38: > FWIW -- this beta builds/tests ok across nearly all Debian ports but 4 Nice! Thanks for testing. > where process unexpectedly terminates for no obvious reason (it > built/tested just fine on my local sparc box): > https://buildd.debian.org/status/package.php?p=cython&suite=experimental > some time I will check WTF (memory? etc) on those intriguing > ports. Could you just give it a rerun? Maybe something or someone external just terminated them accidentally or by purpose or whatever. Stefan From markflorisson88 at gmail.com Mon Aug 27 11:42:24 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Mon, 27 Aug 2012 10:42:24 +0100 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503AC1B3.90807@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503AC1B3.90807@uci.edu> Message-ID: On 27 August 2012 01:39, Christoph Gohlke wrote: > On 8/26/2012 4:08 AM, mark florisson wrote: >> >> On 25 August 2012 03:07, Christoph Gohlke wrote: >>> >>> Hi, >>> >>> >>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>> >>>> >>>> Hi, >>>> >>>> thanks for testing! >>>> >>>> Christoph Gohlke, 24.08.2012 07:20: >>>>> >>>>> >>>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>>> >>>>> 32 bit Python 2.7 works well, only 4 test failures. >>>> >>>> >>>> >>>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>>> build environment? >>> >>> >>> >>> OpenMP is available on my system, and parallel.pyd is linked to the >>> openmp >>> library. The prange tests fail only sometimes. On my system, the prange >>> index is sometimes left at the start (zero) of the range, while the tests >>> expect the index to be left at the stop of the range. According to the >>> Cython prange enhancements webpage "the iterations of the loop body can >>> be >>> executed in any order" . >>> Where >>> does that leave the loop index? >>> >> >> I should be the value from the last iteration, but in my experience >> many compilers have buggy OpenMP implementations. I think your >> compiler doesn't correctly support the lastprivate clause in all >> situations. For instance, test_prange fails (which doesn't have any >> break/return/exceptions), which simply has a lastprivate clause and a >> schedule clause set to 'dynamic'. The test without the schedule clause >> works fine (or maybe it's just luck). Or maybe it doesn't support >> multiple lastprivate() clauses? I'm not entirely sure... It also seems >> the thread limit on your system is 1. >> >> In any case, the generated code for these tests looks correct to me, >> but we've had similar problems before with different compilers... > > > > On my system, the following patch fixes all the lastprivate related test > errors and does not have any side effects on other tests. It removes the > additional initialization of the target index after `#pragma omp parallel`: > > diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py > index 188de3d..5ba0eee 100644 > --- a/Cython/Compiler/Nodes.py > +++ b/Cython/Compiler/Nodes.py > @@ -7627,7 +7627,7 @@ class ParallelRangeNode(ParallelStatNode): > # target index uninitialized > code.putln("if (%(nsteps)s > 0)" % fmt_dict) > code.begin_block() # if block > - code.putln("%(target)s = 0;" % fmt_dict) > self.generate_loop(code, fmt_dict) > code.end_block() # end if block > > I'm down to one remaining test failure on win32-py2.7 > > Christoph > Hey Christoph, Thanks, that's useful. Could you make a pull request? Maybe because the entry to the parallel loop has no barrier the assignment comes after the lastprivate? I'm not sure how that's possible, I would expect the lastprivate to be executed after the barrier by the highest ranking thread, so there would be no race. Maybe the last thread is slow, and doesn't get any iterations, and assigns the firstprivate value? Anyway, if it fixes stuff it's great :) I think the patch should then also remove the firstprivate() clause, since that variable wouldn't be initialized. It was only there to make compiler warnings to away. Mark > > >> >>> >>>> >>>> The other one is the new "initial_file_path" test that fails with this >>>> linker error: >>>> >>>> """ >>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>>> /LIBPATH:X:\Python27\libs >>>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>>> >>>> >>>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>>> >>>> >>>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>>> >>>> LINK : error LNK2001: unresolved external symbol init__init__ >>>> """ >>>> >>>> Maybe the Windows build of distutils is broken here - it seems to assume >>>> the wrong module name for the package module. >>> >>> >>> >>> I think this is an issue with the test. The extension does compile and >>> link >>> outside of the tests. >>> >>> >>> >>>> >>>> I guess compiling package modules is just an overall badly supported >>>> feature in CPython... >>>> >>>> >>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>> during >>>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. >>>>> The >>>>> Windows executive can not identify in which specific module it crashes, >>>>> and >>>>> neither enabling faulthandler nor building with debug symbols gives any >>>>> useful information. Can anyone reproduce this? It seems compiler >>>>> specific >>>>> since Python 3.3, which is using msvc10, does not crash. >>>> >>>> >>>> >>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>> this sorted out, but it's almost impossible to debug something like this >>>> from a distance. >>> >>> >>> >>> >>> Maybe the following simple example is related. It fails (not crash) when >>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 >>> (32 >>> and 64 bit): >>> >>> ``` >>> from cython.view cimport array as cvarray >>> import numpy as np >>> >>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>> >>> cdef int[:, :, ::1] a = narr >>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>> >>> print narr[2:8:2, -4:1:-1, 1:3].shape >>> print b.shape[0], b.shape[1], b.shape[2] >>> ``` >>> >>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >>> >>> >>> >>>> >>>> >>>>> When disabling >>>>> test_slice_assignment, runtests.py completes with many failures. >>>>> >>>>> The results of `runtests.py -v -v` are at >>>>> . >>>> >>>> >>>> >>>> The 64bit output looks so broken that I wonder what went wrong here. I >>>> mean, most of the problems look like this: >>>> >>>> """ >>>> Expected: >>>> Traceback (most recent call last): >>>> TypeError: m() takes at most 2 positional arguments (3 given) >>>> Got: >>>> Traceback (most recent call last): >>>> TypeError: m() takes at most %Id positional argument%s (%Id given) >>>> """ >>>> >>>> I have no idea how that can happen. >>>> >>>> I can see two other problems, one is the linker warning about the module >>>> init function in Py3: >>>> >>>> """ >>>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>>> multiple times; using first specification >>>> """ >>> >>> >>> >>> This is "normal". >>> >>> >>> >>>> >>>> The other one is about mixing Py_ssize_t and int (and some other) types >>>> all >>>> over the place: >>>> >>>> """ >>>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >>>> 'int', possible loss of data >>>> """ >>>> >>>> Some of them look like we'd need an explicit cast in the C code >>>> somewhere, >>>> others might hint at lax type usage in tests. >>>> >>>> There's also this: >>>> >>>> """ >>>> C:\Program Files (x86)\Microsoft Visual Studio >>>> 10.0\VC\INCLUDE\xlocale(323) >>>> : warning C4530: C++ exception handler used, but unwind semantics are >>>> not >>>> enabled. Specify /EHsc >>>> """ >>>> >>>> Stefan >>>> >>> >>> >>> Thanks, >>> >>> Christoph >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> >> >> > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From d.s.seljebotn at astro.uio.no Mon Aug 27 11:55:49 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 27 Aug 2012 11:55:49 +0200 Subject: [Cython] array expressions In-Reply-To: <503B438F.6050504@astro.uio.no> References: <503B438F.6050504@astro.uio.no> Message-ID: <503B4425.3000807@astro.uio.no> On 08/27/2012 11:53 AM, Dag Sverre Seljebotn wrote: > On 08/24/2012 08:40 PM, mark florisson wrote: >> Hey, >> >> Here a pull request for element-wise array expressions for Cython: >> https://github.com/cython/cython/pull/144 >> It includes the IndexNode refactoring branch as well. >> >> This has been the work this last summer for the gsoc, with great >> supervision from Dag, who helped steer the project in a great >> direction to make it reusable (it's partially included in Numba and >> will likely be in Theano in the future, hopefully others as well). I >> also wrote a thesis for my master's, which can be found here >> https://github.com/markflorisson88/minivect/tree/master/thesis, which >> can shed >> some light on some parts of the design and performance aspects. >> Performance graphs can also be found here: >> https://github.com/markflorisson88/minivect/tree/master/bench/graphs >> >> So anyway, how would you prefer dealing with the minivect submodule? >> We could include it verbatim, with any modifications made to minivect >> directly, since we'd have separate git histories. We could >> alternatively make it an optional submodule which is only required >> when actually using array expressions. I like the latter, but anything >> is fine with me really. > > I think I support using a git submodule for now. Not sure about making > it optional (which I assume would make array expression testcases not > run if it minivect is not present so that there's no test failures); we > want to make sure we are forced to do releases and testing right and > include it, and if it is only required to compile code that uses > memoryview expressions users could get confused about there being two > Cython "editions" around. > > Since "git submodule" does link to a specific revision, so there's no > stability concerns over verbatim inclusion. > > How hg-git deals with submodules is worth consideration too though. Another option you didn't mention is to push the responsibility of getting minivect over to end-users; Cython simply tries to do "import minivect". This does have versioning issues though since git will likely not be the tool used to fetch the revision. A lot more pain for those who uses array expressions, but a little less pain for the rest. So it depends on how you weigh the user groups. Realistically, we'd want to depend on LLVM as well down the road for minivect stuff (at least if you want optimal performance), so perhaps opening the can-of-external-dependency-worms should just be done sooner rather than later. Dag From d.s.seljebotn at astro.uio.no Mon Aug 27 11:53:19 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 27 Aug 2012 11:53:19 +0200 Subject: [Cython] array expressions In-Reply-To: References: Message-ID: <503B438F.6050504@astro.uio.no> On 08/24/2012 08:40 PM, mark florisson wrote: > Hey, > > Here a pull request for element-wise array expressions for Cython: > https://github.com/cython/cython/pull/144 > It includes the IndexNode refactoring branch as well. > > This has been the work this last summer for the gsoc, with great > supervision from Dag, who helped steer the project in a great > direction to make it reusable (it's partially included in Numba and > will likely be in Theano in the future, hopefully others as well). I > also wrote a thesis for my master's, which can be found here > https://github.com/markflorisson88/minivect/tree/master/thesis, which > can shed > some light on some parts of the design and performance aspects. > Performance graphs can also be found here: > https://github.com/markflorisson88/minivect/tree/master/bench/graphs > > So anyway, how would you prefer dealing with the minivect submodule? > We could include it verbatim, with any modifications made to minivect > directly, since we'd have separate git histories. We could > alternatively make it an optional submodule which is only required > when actually using array expressions. I like the latter, but anything > is fine with me really. I think I support using a git submodule for now. Not sure about making it optional (which I assume would make array expression testcases not run if it minivect is not present so that there's no test failures); we want to make sure we are forced to do releases and testing right and include it, and if it is only required to compile code that uses memoryview expressions users could get confused about there being two Cython "editions" around. Since "git submodule" does link to a specific revision, so there's no stability concerns over verbatim inclusion. How hg-git deals with submodules is worth consideration too though. Dag From stefan_ml at behnel.de Mon Aug 27 13:06:55 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Aug 2012 13:06:55 +0200 Subject: [Cython] array expressions In-Reply-To: <503B4425.3000807@astro.uio.no> References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> Message-ID: <503B54CF.6090402@behnel.de> Dag Sverre Seljebotn, 27.08.2012 11:55: > On 08/27/2012 11:53 AM, Dag Sverre Seljebotn wrote: >> On 08/24/2012 08:40 PM, mark florisson wrote: >>> Here a pull request for element-wise array expressions for Cython: >>> https://github.com/cython/cython/pull/144 >>> It includes the IndexNode refactoring branch as well. >>> >>> This has been the work this last summer for the gsoc, with great >>> supervision from Dag, who helped steer the project in a great >>> direction to make it reusable (it's partially included in Numba and >>> will likely be in Theano in the future, hopefully others as well). I >>> also wrote a thesis for my master's, which can be found here >>> https://github.com/markflorisson88/minivect/tree/master/thesis, which >>> can shed >>> some light on some parts of the design and performance aspects. >>> Performance graphs can also be found here: >>> https://github.com/markflorisson88/minivect/tree/master/bench/graphs >>> >>> So anyway, how would you prefer dealing with the minivect submodule? >>> We could include it verbatim, with any modifications made to minivect >>> directly, since we'd have separate git histories. We could >>> alternatively make it an optional submodule which is only required >>> when actually using array expressions. I like the latter, but anything >>> is fine with me really. >> >> I think I support using a git submodule for now. Not sure about making >> it optional (which I assume would make array expression testcases not >> run if it minivect is not present so that there's no test failures); we >> want to make sure we are forced to do releases and testing right and >> include it, and if it is only required to compile code that uses >> memoryview expressions users could get confused about there being two >> Cython "editions" around. >> >> Since "git submodule" does link to a specific revision, so there's no >> stability concerns over verbatim inclusion. >> >> How hg-git deals with submodules is worth consideration too though. > > Another option you didn't mention is to push the responsibility of getting > minivect over to end-users; Cython simply tries to do "import minivect". > This does have versioning issues though since git will likely not be the > tool used to fetch the revision. > > A lot more pain for those who uses array expressions, but a little less > pain for the rest. So it depends on how you weigh the user groups. > > Realistically, we'd want to depend on LLVM as well down the road for > minivect stuff (at least if you want optimal performance), so perhaps > opening the can-of-external-dependency-worms should just be done sooner > rather than later. My experience with lxml tells me that it's often better to keep things separate but integrated, instead of shipping them in a big box. As long as it doesn't hurt too much to have separate tools, we should keep it that way. Those who prefer everything in a big box can use distributions like Sage, or use apt. As for versioning, you can set dependency version ranges in distutils (and friends) which are honoured by install tools like pip. That keeps the installation fully automatic (and definitely not "a lot more pain"). That being said, the best way to handle this is to build a well defined interface between the two components and to keep that alive for a while. For Jenkins, we'd set up separate jobs that build the dependencies and then install them from there before running the integration tests. We could even have dedicated integration test jobs that only run the tests that involve the dependency (and potentially more than one version of the dependency). Stefan From markflorisson88 at gmail.com Mon Aug 27 13:13:01 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Mon, 27 Aug 2012 12:13:01 +0100 Subject: [Cython] array expressions In-Reply-To: <503B54CF.6090402@behnel.de> References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> Message-ID: On 27 August 2012 12:06, Stefan Behnel wrote: > Dag Sverre Seljebotn, 27.08.2012 11:55: >> On 08/27/2012 11:53 AM, Dag Sverre Seljebotn wrote: >>> On 08/24/2012 08:40 PM, mark florisson wrote: >>>> Here a pull request for element-wise array expressions for Cython: >>>> https://github.com/cython/cython/pull/144 >>>> It includes the IndexNode refactoring branch as well. >>>> >>>> This has been the work this last summer for the gsoc, with great >>>> supervision from Dag, who helped steer the project in a great >>>> direction to make it reusable (it's partially included in Numba and >>>> will likely be in Theano in the future, hopefully others as well). I >>>> also wrote a thesis for my master's, which can be found here >>>> https://github.com/markflorisson88/minivect/tree/master/thesis, which >>>> can shed >>>> some light on some parts of the design and performance aspects. >>>> Performance graphs can also be found here: >>>> https://github.com/markflorisson88/minivect/tree/master/bench/graphs >>>> >>>> So anyway, how would you prefer dealing with the minivect submodule? >>>> We could include it verbatim, with any modifications made to minivect >>>> directly, since we'd have separate git histories. We could >>>> alternatively make it an optional submodule which is only required >>>> when actually using array expressions. I like the latter, but anything >>>> is fine with me really. >>> >>> I think I support using a git submodule for now. Not sure about making >>> it optional (which I assume would make array expression testcases not >>> run if it minivect is not present so that there's no test failures); we >>> want to make sure we are forced to do releases and testing right and >>> include it, and if it is only required to compile code that uses >>> memoryview expressions users could get confused about there being two >>> Cython "editions" around. >>> >>> Since "git submodule" does link to a specific revision, so there's no >>> stability concerns over verbatim inclusion. >>> >>> How hg-git deals with submodules is worth consideration too though. >> >> Another option you didn't mention is to push the responsibility of getting >> minivect over to end-users; Cython simply tries to do "import minivect". >> This does have versioning issues though since git will likely not be the >> tool used to fetch the revision. >> >> A lot more pain for those who uses array expressions, but a little less >> pain for the rest. So it depends on how you weigh the user groups. >> >> Realistically, we'd want to depend on LLVM as well down the road for >> minivect stuff (at least if you want optimal performance), so perhaps >> opening the can-of-external-dependency-worms should just be done sooner >> rather than later. > > My experience with lxml tells me that it's often better to keep things > separate but integrated, instead of shipping them in a big box. As long as > it doesn't hurt too much to have separate tools, we should keep it that > way. Those who prefer everything in a big box can use distributions like > Sage, or use apt. > > As for versioning, you can set dependency version ranges in distutils (and > friends) which are honoured by install tools like pip. That keeps the > installation fully automatic (and definitely not "a lot more pain"). That > being said, the best way to handle this is to build a well defined > interface between the two components and to keep that alive for a while. > > For Jenkins, we'd set up separate jobs that build the dependencies and then > install them from there before running the integration tests. We could even > have dedicated integration test jobs that only run the tests that involve > the dependency (and potentially more than one version of the dependency). > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Oh great, that works for me as well. I thought Cython's take was to avoid dependencies by shipping everything (like Plex and Tempita). I'll make minivect distributable, then. From stefan_ml at behnel.de Mon Aug 27 13:40:03 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Aug 2012 13:40:03 +0200 Subject: [Cython] array expressions In-Reply-To: References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> Message-ID: <503B5C93.3080309@behnel.de> mark florisson, 27.08.2012 13:13: > On 27 August 2012 12:06, Stefan Behnel wrote: >> My experience with lxml tells me that it's often better to keep things >> separate but integrated, instead of shipping them in a big box. As long as >> it doesn't hurt too much to have separate tools, we should keep it that >> way. Those who prefer everything in a big box can use distributions like >> Sage, or use apt. >> >> As for versioning, you can set dependency version ranges in distutils (and >> friends) which are honoured by install tools like pip. That keeps the >> installation fully automatic (and definitely not "a lot more pain"). That >> being said, the best way to handle this is to build a well defined >> interface between the two components and to keep that alive for a while. >> >> For Jenkins, we'd set up separate jobs that build the dependencies and then >> install them from there before running the integration tests. We could even >> have dedicated integration test jobs that only run the tests that involve >> the dependency (and potentially more than one version of the dependency). > > Oh great, that works for me as well. I thought Cython's take was to > avoid dependencies by shipping everything (like Plex and Tempita). Those are core dependencies. Plex is actually modified IIRC, and even compiled. Tempita was specifically chosen because it's so small and can be shipped. > I'll make minivect distributable, then. I think the difference here is that it's an optional dependency. It makes sense independently as well when used with other tools. And from my distant view, it seems like it could deserve its own project. The code tree looks a bit disorganised currently, with a lot of modules at the top level. I guess that's what you meant when you said "make it distributable". BTW, have you been in touch with the PyPy people about this? They should have implemented their own vectorisation for what they call numpypy. Maybe your tool would be of interest for them as well? Stefan From markflorisson88 at gmail.com Mon Aug 27 13:52:45 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Mon, 27 Aug 2012 12:52:45 +0100 Subject: [Cython] array expressions In-Reply-To: <503B5C93.3080309@behnel.de> References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> <503B5C93.3080309@behnel.de> Message-ID: On 27 August 2012 12:40, Stefan Behnel wrote: > mark florisson, 27.08.2012 13:13: >> On 27 August 2012 12:06, Stefan Behnel wrote: >>> My experience with lxml tells me that it's often better to keep things >>> separate but integrated, instead of shipping them in a big box. As long as >>> it doesn't hurt too much to have separate tools, we should keep it that >>> way. Those who prefer everything in a big box can use distributions like >>> Sage, or use apt. >>> >>> As for versioning, you can set dependency version ranges in distutils (and >>> friends) which are honoured by install tools like pip. That keeps the >>> installation fully automatic (and definitely not "a lot more pain"). That >>> being said, the best way to handle this is to build a well defined >>> interface between the two components and to keep that alive for a while. >>> >>> For Jenkins, we'd set up separate jobs that build the dependencies and then >>> install them from there before running the integration tests. We could even >>> have dedicated integration test jobs that only run the tests that involve >>> the dependency (and potentially more than one version of the dependency). >> >> Oh great, that works for me as well. I thought Cython's take was to >> avoid dependencies by shipping everything (like Plex and Tempita). > > Those are core dependencies. Plex is actually modified IIRC, and even > compiled. Tempita was specifically chosen because it's so small and can be > shipped. > > >> I'll make minivect distributable, then. > > I think the difference here is that it's an optional dependency. It makes > sense independently as well when used with other tools. And from my distant > view, it seems like it could deserve its own project. > > The code tree looks a bit disorganised currently, with a lot of modules at > the top level. I guess that's what you meant when you said "make it > distributable". Indeed, the idea was that it would be included verbatim in projects/used as a submodule, since it'd be small enough to include and would always be stable and integrated into the main project. That basically means the entire thing would be the package. It'd be kind of nice to allow it both ways, i.e. install as a package, and make it convenient as a submodule. I think that can be managed. E.g. I can see situations where you need to fix a little bug, or make a small change where you don't really want (to wait for) a new release of minivect, but you do want that exact commit for the release of the main project. > BTW, have you been in touch with the PyPy people about this? They should > have implemented their own vectorisation for what they call numpypy. Maybe > your tool would be of interest for them as well? I was thinking about it, not yet. I think it'd have to be adapted somehow, since it uses llvmpy. It's also not entirely featureful yet (e.g. no reductions), since most of the focus has been on performance. But I'll raise the issue on the ML, thanks :) > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Mon Aug 27 14:05:39 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Aug 2012 14:05:39 +0200 Subject: [Cython] array expressions In-Reply-To: References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> <503B5C93.3080309@behnel.de> Message-ID: <503B6293.6020103@behnel.de> mark florisson, 27.08.2012 13:52: > On 27 August 2012 12:40, Stefan Behnel wrote: >> mark florisson, 27.08.2012 13:13: >>> On 27 August 2012 12:06, Stefan Behnel wrote: >>>> My experience with lxml tells me that it's often better to keep things >>>> separate but integrated, instead of shipping them in a big box. As long as >>>> it doesn't hurt too much to have separate tools, we should keep it that >>>> way. Those who prefer everything in a big box can use distributions like >>>> Sage, or use apt. >>>> >>>> As for versioning, you can set dependency version ranges in distutils (and >>>> friends) which are honoured by install tools like pip. That keeps the >>>> installation fully automatic (and definitely not "a lot more pain"). That >>>> being said, the best way to handle this is to build a well defined >>>> interface between the two components and to keep that alive for a while. >>>> >>>> For Jenkins, we'd set up separate jobs that build the dependencies and then >>>> install them from there before running the integration tests. We could even >>>> have dedicated integration test jobs that only run the tests that involve >>>> the dependency (and potentially more than one version of the dependency). >>> >>> Oh great, that works for me as well. I thought Cython's take was to >>> avoid dependencies by shipping everything (like Plex and Tempita). >> >> Those are core dependencies. Plex is actually modified IIRC, and even >> compiled. Tempita was specifically chosen because it's so small and can be >> shipped. >> >> >>> I'll make minivect distributable, then. >> >> I think the difference here is that it's an optional dependency. It makes >> sense independently as well when used with other tools. And from my distant >> view, it seems like it could deserve its own project. >> >> The code tree looks a bit disorganised currently, with a lot of modules at >> the top level. I guess that's what you meant when you said "make it >> distributable". > > Indeed, the idea was that it would be included verbatim in > projects/used as a submodule, since it'd be small enough to include > and would always be stable and integrated into the main project. That > basically means the entire thing would be the package. It'd be kind of > nice to allow it both ways, i.e. install as a package, and make it > convenient as a submodule. I think that can be managed. I'd just move it into a package anyway and let projects that want to use it as a git submodule move it to an appropriate place during installation by appropriately configuring their project specific distutils setup. > E.g. I can see situations where you need to fix a little bug, or make > a small change where you don't really want (to wait for) a new release > of minivect, but you do want that exact commit for the release of the > main project. Doesn't that apply to more or less all code out there? Stefan From d.s.seljebotn at astro.uio.no Mon Aug 27 14:45:18 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 27 Aug 2012 14:45:18 +0200 Subject: [Cython] array expressions In-Reply-To: <503B6293.6020103@behnel.de> References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> <503B5C93.3080309@behnel.de> <503B6293.6020103@behnel.de> Message-ID: <503B6BDE.8060706@astro.uio.no> On 08/27/2012 02:05 PM, Stefan Behnel wrote: > mark florisson, 27.08.2012 13:52: >> On 27 August 2012 12:40, Stefan Behnel wrote: >>> mark florisson, 27.08.2012 13:13: >>>> On 27 August 2012 12:06, Stefan Behnel wrote: >>>>> My experience with lxml tells me that it's often better to keep things >>>>> separate but integrated, instead of shipping them in a big box. As long as >>>>> it doesn't hurt too much to have separate tools, we should keep it that >>>>> way. Those who prefer everything in a big box can use distributions like >>>>> Sage, or use apt. >>>>> >>>>> As for versioning, you can set dependency version ranges in distutils (and >>>>> friends) which are honoured by install tools like pip. That keeps the >>>>> installation fully automatic (and definitely not "a lot more pain"). That >>>>> being said, the best way to handle this is to build a well defined >>>>> interface between the two components and to keep that alive for a while. >>>>> >>>>> For Jenkins, we'd set up separate jobs that build the dependencies and then >>>>> install them from there before running the integration tests. We could even >>>>> have dedicated integration test jobs that only run the tests that involve >>>>> the dependency (and potentially more than one version of the dependency). >>>> >>>> Oh great, that works for me as well. I thought Cython's take was to >>>> avoid dependencies by shipping everything (like Plex and Tempita). >>> >>> Those are core dependencies. Plex is actually modified IIRC, and even >>> compiled. Tempita was specifically chosen because it's so small and can be >>> shipped. >>> >>> >>>> I'll make minivect distributable, then. >>> >>> I think the difference here is that it's an optional dependency. It makes >>> sense independently as well when used with other tools. And from my distant >>> view, it seems like it could deserve its own project. >>> >>> The code tree looks a bit disorganised currently, with a lot of modules at >>> the top level. I guess that's what you meant when you said "make it >>> distributable". >> >> Indeed, the idea was that it would be included verbatim in >> projects/used as a submodule, since it'd be small enough to include >> and would always be stable and integrated into the main project. That >> basically means the entire thing would be the package. It'd be kind of >> nice to allow it both ways, i.e. install as a package, and make it >> convenient as a submodule. I think that can be managed. > > I'd just move it into a package anyway and let projects that want to use it > as a git submodule move it to an appropriate place during installation by > appropriately configuring their project specific distutils setup. > > >> E.g. I can see situations where you need to fix a little bug, or make >> a small change where you don't really want (to wait for) a new release >> of minivect, but you do want that exact commit for the release of the >> main project. > > Doesn't that apply to more or less all code out there? Most package dependencies are *a lot* more loosely coupled that Cython (w/some features)<->minivect. Since minivect works with an AST and all. Dag From d.s.seljebotn at astro.uio.no Mon Aug 27 14:47:46 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 27 Aug 2012 14:47:46 +0200 Subject: [Cython] array expressions In-Reply-To: <503B6BDE.8060706@astro.uio.no> References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> <503B5C93.3080309@behnel.de> <503B6293.6020103@behnel.de> <503B6BDE.8060706@astro.uio.no> Message-ID: <503B6C72.9020407@astro.uio.no> On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote: > On 08/27/2012 02:05 PM, Stefan Behnel wrote: >> mark florisson, 27.08.2012 13:52: >>> On 27 August 2012 12:40, Stefan Behnel wrote: >>>> mark florisson, 27.08.2012 13:13: >>>>> On 27 August 2012 12:06, Stefan Behnel wrote: >>>>>> My experience with lxml tells me that it's often better to keep >>>>>> things >>>>>> separate but integrated, instead of shipping them in a big box. As >>>>>> long as >>>>>> it doesn't hurt too much to have separate tools, we should keep it >>>>>> that >>>>>> way. Those who prefer everything in a big box can use >>>>>> distributions like >>>>>> Sage, or use apt. >>>>>> >>>>>> As for versioning, you can set dependency version ranges in >>>>>> distutils (and >>>>>> friends) which are honoured by install tools like pip. That keeps the >>>>>> installation fully automatic (and definitely not "a lot more >>>>>> pain"). That >>>>>> being said, the best way to handle this is to build a well defined >>>>>> interface between the two components and to keep that alive for a >>>>>> while. >>>>>> >>>>>> For Jenkins, we'd set up separate jobs that build the dependencies >>>>>> and then >>>>>> install them from there before running the integration tests. We >>>>>> could even >>>>>> have dedicated integration test jobs that only run the tests that >>>>>> involve >>>>>> the dependency (and potentially more than one version of the >>>>>> dependency). >>>>> >>>>> Oh great, that works for me as well. I thought Cython's take was to >>>>> avoid dependencies by shipping everything (like Plex and Tempita). >>>> >>>> Those are core dependencies. Plex is actually modified IIRC, and even >>>> compiled. Tempita was specifically chosen because it's so small and >>>> can be >>>> shipped. >>>> >>>> >>>>> I'll make minivect distributable, then. >>>> >>>> I think the difference here is that it's an optional dependency. It >>>> makes >>>> sense independently as well when used with other tools. And from my >>>> distant >>>> view, it seems like it could deserve its own project. >>>> >>>> The code tree looks a bit disorganised currently, with a lot of >>>> modules at >>>> the top level. I guess that's what you meant when you said "make it >>>> distributable". >>> >>> Indeed, the idea was that it would be included verbatim in >>> projects/used as a submodule, since it'd be small enough to include >>> and would always be stable and integrated into the main project. That >>> basically means the entire thing would be the package. It'd be kind of >>> nice to allow it both ways, i.e. install as a package, and make it >>> convenient as a submodule. I think that can be managed. >> >> I'd just move it into a package anyway and let projects that want to >> use it >> as a git submodule move it to an appropriate place during installation by >> appropriately configuring their project specific distutils setup. >> >> >>> E.g. I can see situations where you need to fix a little bug, or make >>> a small change where you don't really want (to wait for) a new release >>> of minivect, but you do want that exact commit for the release of the >>> main project. >> >> Doesn't that apply to more or less all code out there? > > Most package dependencies are *a lot* more loosely coupled that Cython > (w/some features)<->minivect. Since minivect works with an AST and all. I think there's a case for having minivect as a git submodule for convenient development purposes (so that you can easily sort of refactor Cython and minivect together), but that's only a development mechanism. I.e., Cython first tries to import 'Cython.minivect' (which could be a symlink into a git submodule) and if that fails 'minivect', the latter would be the normal end-user mode because Cython doesn't bundle minivect. Dag From stefan_ml at behnel.de Mon Aug 27 14:53:49 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Aug 2012 14:53:49 +0200 Subject: [Cython] array expressions In-Reply-To: <503B6C72.9020407@astro.uio.no> References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> <503B5C93.3080309@behnel.de> <503B6293.6020103@behnel.de> <503B6BDE.8060706@astro.uio.no> <503B6C72.9020407@astro.uio.no> Message-ID: <503B6DDD.1090302@behnel.de> Dag Sverre Seljebotn, 27.08.2012 14:47: > On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote: >> On 08/27/2012 02:05 PM, Stefan Behnel wrote: >>> mark florisson, 27.08.2012 13:52: >>>> On 27 August 2012 12:40, Stefan Behnel wrote: >>>>> The code tree looks a bit disorganised currently, with a lot of >>>>> modules at >>>>> the top level. I guess that's what you meant when you said "make it >>>>> distributable". >>>> >>>> Indeed, the idea was that it would be included verbatim in >>>> projects/used as a submodule, since it'd be small enough to include >>>> and would always be stable and integrated into the main project. That >>>> basically means the entire thing would be the package. It'd be kind of >>>> nice to allow it both ways, i.e. install as a package, and make it >>>> convenient as a submodule. I think that can be managed. >>> >>> I'd just move it into a package anyway and let projects that want to >>> use it >>> as a git submodule move it to an appropriate place during installation by >>> appropriately configuring their project specific distutils setup. >>> >>>> E.g. I can see situations where you need to fix a little bug, or make >>>> a small change where you don't really want (to wait for) a new release >>>> of minivect, but you do want that exact commit for the release of the >>>> main project. >>> >>> Doesn't that apply to more or less all code out there? >> >> Most package dependencies are *a lot* more loosely coupled that Cython >> (w/some features)<->minivect. Since minivect works with an AST and all. > > I think there's a case for having minivect as a git submodule for > convenient development purposes (so that you can easily sort of refactor > Cython and minivect together), but that's only a development mechanism. > > I.e., Cython first tries to import 'Cython.minivect' (which could be a > symlink into a git submodule) and if that fails 'minivect', the latter > would be the normal end-user mode because Cython doesn't bundle minivect. python setup.py develop (or the equivalent options for pip and easy_install) Stefan From cgohlke at uci.edu Mon Aug 27 18:05:40 2012 From: cgohlke at uci.edu (Christoph Gohlke) Date: Mon, 27 Aug 2012 09:05:40 -0700 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503AC1B3.90807@uci.edu> Message-ID: <503B9AD4.2000707@uci.edu> On 8/27/2012 2:42 AM, mark florisson wrote: > On 27 August 2012 01:39, Christoph Gohlke wrote: >> On 8/26/2012 4:08 AM, mark florisson wrote: >>> >>> On 25 August 2012 03:07, Christoph Gohlke wrote: >>>> >>>> Hi, >>>> >>>> >>>> On 8/24/2012 12:43 PM, Stefan Behnel wrote: >>>>> >>>>> >>>>> Hi, >>>>> >>>>> thanks for testing! >>>>> >>>>> Christoph Gohlke, 24.08.2012 07:20: >>>>>> >>>>>> >>>>>> I tested Cython-0.17b3 on Windows 7 with Visual Studio compilers. >>>>>> >>>>>> 32 bit Python 2.7 works well, only 4 test failures. >>>>> >>>>> >>>>> >>>>> Three of those errors are in OpenMP tests - is OpenMP supported in your >>>>> build environment? >>>> >>>> >>>> >>>> OpenMP is available on my system, and parallel.pyd is linked to the >>>> openmp >>>> library. The prange tests fail only sometimes. On my system, the prange >>>> index is sometimes left at the start (zero) of the range, while the tests >>>> expect the index to be left at the stop of the range. According to the >>>> Cython prange enhancements webpage "the iterations of the loop body can >>>> be >>>> executed in any order" . >>>> Where >>>> does that leave the loop index? >>>> >>> >>> I should be the value from the last iteration, but in my experience >>> many compilers have buggy OpenMP implementations. I think your >>> compiler doesn't correctly support the lastprivate clause in all >>> situations. For instance, test_prange fails (which doesn't have any >>> break/return/exceptions), which simply has a lastprivate clause and a >>> schedule clause set to 'dynamic'. The test without the schedule clause >>> works fine (or maybe it's just luck). Or maybe it doesn't support >>> multiple lastprivate() clauses? I'm not entirely sure... It also seems >>> the thread limit on your system is 1. >>> >>> In any case, the generated code for these tests looks correct to me, >>> but we've had similar problems before with different compilers... >> >> >> >> On my system, the following patch fixes all the lastprivate related test >> errors and does not have any side effects on other tests. It removes the >> additional initialization of the target index after `#pragma omp parallel`: >> >> diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py >> index 188de3d..5ba0eee 100644 >> --- a/Cython/Compiler/Nodes.py >> +++ b/Cython/Compiler/Nodes.py >> @@ -7627,7 +7627,7 @@ class ParallelRangeNode(ParallelStatNode): >> # target index uninitialized >> code.putln("if (%(nsteps)s > 0)" % fmt_dict) >> code.begin_block() # if block >> - code.putln("%(target)s = 0;" % fmt_dict) >> self.generate_loop(code, fmt_dict) >> code.end_block() # end if block >> >> I'm down to one remaining test failure on win32-py2.7 >> >> Christoph >> > > Hey Christoph, > > Thanks, that's useful. Could you make a pull request? Maybe because > the entry to the parallel loop has no barrier the assignment comes > after the lastprivate? I'm not sure how that's possible, I would > expect the lastprivate to be executed after the barrier by the highest > ranking thread, so there would be no race. Maybe the last thread is > slow, and doesn't get any iterations, and assigns the firstprivate > value? Anyway, if it fixes stuff it's great :) > > I think the patch should then also remove the firstprivate() clause, > since that variable wouldn't be initialized. It was only there to make > compiler warnings to away. > > Mark I opened a pull request. Let's discuss it there. Christoph >> >> >>> >>>> >>>>> >>>>> The other one is the new "initial_file_path" test that fails with this >>>>> linker error: >>>>> >>>>> """ >>>>> C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL >>>>> /nologo /INCREMENTAL:NO /INCREMENTAL:YES /DEBUG >>>>> /LIBPATH:X:\Python27\libs >>>>> /LIBPATH:X:\Python27\PCbuild /EXPORT:init__init__ >>>>> build\temp.win32-2.7\Release\my_test_package\__init__.obj >>>>> >>>>> >>>>> /OUT:\Cython-0.17b3\BUILD\run\initial_file_path\my_test_package\__init__.pyd >>>>> /IMPLIB:build\temp.win32-2.7\Release\my_test_package\__init__.lib >>>>> >>>>> >>>>> /MANIFESTFILE:build\temp.win32-2.7\Release\my_test_package\__init__.pyd.manifest >>>>> >>>>> LINK : error LNK2001: unresolved external symbol init__init__ >>>>> """ >>>>> >>>>> Maybe the Windows build of distutils is broken here - it seems to assume >>>>> the wrong module name for the package module. >>>> >>>> >>>> >>>> I think this is an issue with the test. The extension does compile and >>>> link >>>> outside of the tests. >>>> >>>> >>>> >>>>> >>>>> I guess compiling package modules is just an overall badly supported >>>>> feature in CPython... >>>>> >>>>> >>>>>> On 64 bit Python 2.7 and 3.2 with msvc9 compiler, python.exe crashes >>>>>> during >>>>>> `test_slice_assignment (memslice.__test__)`. I tested two computers. >>>>>> The >>>>>> Windows executive can not identify in which specific module it crashes, >>>>>> and >>>>>> neither enabling faulthandler nor building with debug symbols gives any >>>>>> useful information. Can anyone reproduce this? It seems compiler >>>>>> specific >>>>>> since Python 3.3, which is using msvc10, does not crash. >>>>> >>>>> >>>>> >>>>> Hmm, yes, sounds like a problem with the compiler. Would be good to get >>>>> this sorted out, but it's almost impossible to debug something like this >>>>> from a distance. >>>> >>>> >>>> >>>> >>>> Maybe the following simple example is related. It fails (not crash) when >>>> compiled with 64 bit msvc9, but does work with 32 bit msvc9 and msvc10 >>>> (32 >>>> and 64 bit): >>>> >>>> ``` >>>> from cython.view cimport array as cvarray >>>> import numpy as np >>>> >>>> narr = np.arange(8 * 14 * 11).reshape((8, 14, 11)) >>>> >>>> cdef int[:, :, ::1] a = narr >>>> cdef int[:, :, :] b = a[2:8:2, -4:1:-1, 1:3] >>>> >>>> print narr[2:8:2, -4:1:-1, 1:3].shape >>>> print b.shape[0], b.shape[1], b.shape[2] >>>> ``` >>>> >>>> On win-amd64-py2.x the shape of b is (3, 9, 3) but it should be (3, 9, 2) >>>> >>>> >>>> >>>>> >>>>> >>>>>> When disabling >>>>>> test_slice_assignment, runtests.py completes with many failures. >>>>>> >>>>>> The results of `runtests.py -v -v` are at >>>>>> . >>>>> >>>>> >>>>> >>>>> The 64bit output looks so broken that I wonder what went wrong here. I >>>>> mean, most of the problems look like this: >>>>> >>>>> """ >>>>> Expected: >>>>> Traceback (most recent call last): >>>>> TypeError: m() takes at most 2 positional arguments (3 given) >>>>> Got: >>>>> Traceback (most recent call last): >>>>> TypeError: m() takes at most %Id positional argument%s (%Id given) >>>>> """ >>>>> >>>>> I have no idea how that can happen. >>>>> >>>>> I can see two other problems, one is the linker warning about the module >>>>> init function in Py3: >>>>> >>>>> """ >>>>> bufaccess.obj : warning LNK4197: export 'PyInit_bufaccess' specified >>>>> multiple times; using first specification >>>>> """ >>>> >>>> >>>> >>>> This is "normal". >>>> >>>> >>>> >>>>> >>>>> The other one is about mixing Py_ssize_t and int (and some other) types >>>>> all >>>>> over the place: >>>>> >>>>> """ >>>>> bufaccess.c(3033) : warning C4244: '=' : conversion from 'Py_ssize_t' to >>>>> 'int', possible loss of data >>>>> """ >>>>> >>>>> Some of them look like we'd need an explicit cast in the C code >>>>> somewhere, >>>>> others might hint at lax type usage in tests. >>>>> >>>>> There's also this: >>>>> >>>>> """ >>>>> C:\Program Files (x86)\Microsoft Visual Studio >>>>> 10.0\VC\INCLUDE\xlocale(323) >>>>> : warning C4530: C++ exception handler used, but unwind semantics are >>>>> not >>>>> enabled. Specify /EHsc >>>>> """ >>>>> >>>>> Stefan >>>>> >>>> >>>> >>>> Thanks, >>>> >>>> Christoph >>>> >>>> _______________________________________________ >>>> cython-devel mailing list >>>> cython-devel at python.org >>>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/mailman/listinfo/cython-devel >>> >>> >>> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > From robertwb at gmail.com Mon Aug 27 20:07:26 2012 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 27 Aug 2012 11:07:26 -0700 Subject: [Cython] array expressions In-Reply-To: <503B6C72.9020407@astro.uio.no> References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> <503B5C93.3080309@behnel.de> <503B6293.6020103@behnel.de> <503B6BDE.8060706@astro.uio.no> <503B6C72.9020407@astro.uio.no> Message-ID: On Mon, Aug 27, 2012 at 5:47 AM, Dag Sverre Seljebotn wrote: > On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote: >> >> On 08/27/2012 02:05 PM, Stefan Behnel wrote: >>> >>> mark florisson, 27.08.2012 13:52: >>>> >>>> On 27 August 2012 12:40, Stefan Behnel wrote: >>>>> >>>>> mark florisson, 27.08.2012 13:13: >>>>>> >>>>>> On 27 August 2012 12:06, Stefan Behnel wrote: >>>>>>> >>>>>>> My experience with lxml tells me that it's often better to keep >>>>>>> things >>>>>>> separate but integrated, instead of shipping them in a big box. As >>>>>>> long as >>>>>>> it doesn't hurt too much to have separate tools, we should keep it >>>>>>> that >>>>>>> way. Those who prefer everything in a big box can use >>>>>>> distributions like >>>>>>> Sage, or use apt. >>>>>>> >>>>>>> As for versioning, you can set dependency version ranges in >>>>>>> distutils (and >>>>>>> friends) which are honoured by install tools like pip. That keeps the >>>>>>> installation fully automatic (and definitely not "a lot more >>>>>>> pain"). That >>>>>>> being said, the best way to handle this is to build a well defined >>>>>>> interface between the two components and to keep that alive for a >>>>>>> while. >>>>>>> >>>>>>> For Jenkins, we'd set up separate jobs that build the dependencies >>>>>>> and then >>>>>>> install them from there before running the integration tests. We >>>>>>> could even >>>>>>> have dedicated integration test jobs that only run the tests that >>>>>>> involve >>>>>>> the dependency (and potentially more than one version of the >>>>>>> dependency). >>>>>> >>>>>> >>>>>> Oh great, that works for me as well. I thought Cython's take was to >>>>>> avoid dependencies by shipping everything (like Plex and Tempita). >>>>> >>>>> >>>>> Those are core dependencies. Plex is actually modified IIRC, and even >>>>> compiled. Tempita was specifically chosen because it's so small and >>>>> can be >>>>> shipped. >>>>> >>>>> >>>>>> I'll make minivect distributable, then. >>>>> >>>>> >>>>> I think the difference here is that it's an optional dependency. It >>>>> makes >>>>> sense independently as well when used with other tools. And from my >>>>> distant >>>>> view, it seems like it could deserve its own project. >>>>> >>>>> The code tree looks a bit disorganised currently, with a lot of >>>>> modules at >>>>> the top level. I guess that's what you meant when you said "make it >>>>> distributable". >>>> >>>> >>>> Indeed, the idea was that it would be included verbatim in >>>> projects/used as a submodule, since it'd be small enough to include >>>> and would always be stable and integrated into the main project. That >>>> basically means the entire thing would be the package. It'd be kind of >>>> nice to allow it both ways, i.e. install as a package, and make it >>>> convenient as a submodule. I think that can be managed. >>> >>> >>> I'd just move it into a package anyway and let projects that want to >>> use it >>> as a git submodule move it to an appropriate place during installation by >>> appropriately configuring their project specific distutils setup. >>> >>> >>>> E.g. I can see situations where you need to fix a little bug, or make >>>> a small change where you don't really want (to wait for) a new release >>>> of minivect, but you do want that exact commit for the release of the >>>> main project. >>> >>> >>> Doesn't that apply to more or less all code out there? >> >> >> Most package dependencies are *a lot* more loosely coupled that Cython >> (w/some features)<->minivect. Since minivect works with an AST and all. > > > I think there's a case for having minivect as a git submodule for convenient > development purposes (so that you can easily sort of refactor Cython and > minivect together), but that's only a development mechanism. > > I.e., Cython first tries to import 'Cython.minivect' (which could be a > symlink into a git submodule) and if that fails 'minivect', the latter would > be the normal end-user mode because Cython doesn't bundle minivect. +1, this seems like the best option to me. - Robert From d.s.seljebotn at astro.uio.no Mon Aug 27 20:16:46 2012 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Mon, 27 Aug 2012 20:16:46 +0200 Subject: [Cython] array expressions In-Reply-To: References: <503B438F.6050504@astro.uio.no> <503B4425.3000807@astro.uio.no> <503B54CF.6090402@behnel.de> <503B5C93.3080309@behnel.de> <503B6293.6020103@behnel.de> <503B6BDE.8060706@astro.uio.no> <503B6C72.9020407@astro.uio.no> Message-ID: <503BB98E.5050204@astro.uio.no> On 08/27/2012 08:07 PM, Robert Bradshaw wrote: > On Mon, Aug 27, 2012 at 5:47 AM, Dag Sverre Seljebotn > wrote: >> On 08/27/2012 02:45 PM, Dag Sverre Seljebotn wrote: >>> >>> On 08/27/2012 02:05 PM, Stefan Behnel wrote: >>>> >>>> mark florisson, 27.08.2012 13:52: >>>>> >>>>> On 27 August 2012 12:40, Stefan Behnel wrote: >>>>>> >>>>>> mark florisson, 27.08.2012 13:13: >>>>>>> >>>>>>> On 27 August 2012 12:06, Stefan Behnel wrote: >>>>>>>> >>>>>>>> My experience with lxml tells me that it's often better to keep >>>>>>>> things >>>>>>>> separate but integrated, instead of shipping them in a big box. As >>>>>>>> long as >>>>>>>> it doesn't hurt too much to have separate tools, we should keep it >>>>>>>> that >>>>>>>> way. Those who prefer everything in a big box can use >>>>>>>> distributions like >>>>>>>> Sage, or use apt. >>>>>>>> >>>>>>>> As for versioning, you can set dependency version ranges in >>>>>>>> distutils (and >>>>>>>> friends) which are honoured by install tools like pip. That keeps the >>>>>>>> installation fully automatic (and definitely not "a lot more >>>>>>>> pain"). That >>>>>>>> being said, the best way to handle this is to build a well defined >>>>>>>> interface between the two components and to keep that alive for a >>>>>>>> while. >>>>>>>> >>>>>>>> For Jenkins, we'd set up separate jobs that build the dependencies >>>>>>>> and then >>>>>>>> install them from there before running the integration tests. We >>>>>>>> could even >>>>>>>> have dedicated integration test jobs that only run the tests that >>>>>>>> involve >>>>>>>> the dependency (and potentially more than one version of the >>>>>>>> dependency). >>>>>>> >>>>>>> >>>>>>> Oh great, that works for me as well. I thought Cython's take was to >>>>>>> avoid dependencies by shipping everything (like Plex and Tempita). >>>>>> >>>>>> >>>>>> Those are core dependencies. Plex is actually modified IIRC, and even >>>>>> compiled. Tempita was specifically chosen because it's so small and >>>>>> can be >>>>>> shipped. >>>>>> >>>>>> >>>>>>> I'll make minivect distributable, then. >>>>>> >>>>>> >>>>>> I think the difference here is that it's an optional dependency. It >>>>>> makes >>>>>> sense independently as well when used with other tools. And from my >>>>>> distant >>>>>> view, it seems like it could deserve its own project. >>>>>> >>>>>> The code tree looks a bit disorganised currently, with a lot of >>>>>> modules at >>>>>> the top level. I guess that's what you meant when you said "make it >>>>>> distributable". >>>>> >>>>> >>>>> Indeed, the idea was that it would be included verbatim in >>>>> projects/used as a submodule, since it'd be small enough to include >>>>> and would always be stable and integrated into the main project. That >>>>> basically means the entire thing would be the package. It'd be kind of >>>>> nice to allow it both ways, i.e. install as a package, and make it >>>>> convenient as a submodule. I think that can be managed. >>>> >>>> >>>> I'd just move it into a package anyway and let projects that want to >>>> use it >>>> as a git submodule move it to an appropriate place during installation by >>>> appropriately configuring their project specific distutils setup. >>>> >>>> >>>>> E.g. I can see situations where you need to fix a little bug, or make >>>>> a small change where you don't really want (to wait for) a new release >>>>> of minivect, but you do want that exact commit for the release of the >>>>> main project. >>>> >>>> >>>> Doesn't that apply to more or less all code out there? >>> >>> >>> Most package dependencies are *a lot* more loosely coupled that Cython >>> (w/some features)<->minivect. Since minivect works with an AST and all. >> >> >> I think there's a case for having minivect as a git submodule for convenient >> development purposes (so that you can easily sort of refactor Cython and >> minivect together), but that's only a development mechanism. >> >> I.e., Cython first tries to import 'Cython.minivect' (which could be a >> symlink into a git submodule) and if that fails 'minivect', the latter would >> be the normal end-user mode because Cython doesn't bundle minivect. > > +1, this seems like the best option to me. I'm not convinced by "setup.py develop" either (I tend to keep distutils out of my workflow whenever I can). Dag From stefan_ml at behnel.de Tue Aug 28 19:00:26 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 28 Aug 2012 19:00:26 +0200 Subject: [Cython] [cython-users] Re: Cython 0.17 beta 3 released - release candidate In-Reply-To: References: <503672BF.5070702@behnel.de> Message-ID: <503CF92A.5050403@behnel.de> Neal Becker, 28.08.2012 18:42: > Neal Becker wrote: > >> All built/tested OK on fedora (devel). >> >> Build log is here: >> http://kojipkgs.fedoraproject.org//work/tasks/8252/4428252/build.log >> >> Should I be concerened about these excluded tests? >> >> Ran 6865 tests in 1601.636s >> OK >> Following tests excluded because of missing dependencies on your system: >> memoryview.memoryviewattrs >> memoryview.numpy_memoryview >> run.numpy_ValueError_T172 >> run.numpy_bufacc_T155 >> run.numpy_cimport >> run.numpy_parallel >> run.numpy_test > > OK, this time including numpy in the build, it appears all tests passed OK. Thanks for testing! Stefan From stefan_ml at behnel.de Wed Aug 29 22:20:03 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 29 Aug 2012 22:20:03 +0200 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503B9AD4.2000707@uci.edu> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503AC1B3.90807@uci.edu> <503B9AD4.2000707@uci.edu> Message-ID: <503E7973.9030803@behnel.de> Christoph Gohlke, 27.08.2012 18:05: > On 8/27/2012 2:42 AM, mark florisson wrote: >> On 27 August 2012 01:39, Christoph Gohlke wrote: >>> On my system, the following patch fixes all the lastprivate related test >>> errors and does not have any side effects on other tests. It removes the >>> additional initialization of the target index after `#pragma omp parallel`: >>> >>> diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py >>> index 188de3d..5ba0eee 100644 >>> --- a/Cython/Compiler/Nodes.py >>> +++ b/Cython/Compiler/Nodes.py >>> @@ -7627,7 +7627,7 @@ class ParallelRangeNode(ParallelStatNode): >>> # target index uninitialized >>> code.putln("if (%(nsteps)s > 0)" % fmt_dict) >>> code.begin_block() # if block >>> - code.putln("%(target)s = 0;" % fmt_dict) >>> self.generate_loop(code, fmt_dict) >>> code.end_block() # end if block >>> >>> I'm down to one remaining test failure on win32-py2.7 >> >> Thanks, that's useful. Could you make a pull request? Maybe because >> the entry to the parallel loop has no barrier the assignment comes >> after the lastprivate? I'm not sure how that's possible, I would >> expect the lastprivate to be executed after the barrier by the highest >> ranking thread, so there would be no race. Maybe the last thread is >> slow, and doesn't get any iterations, and assigns the firstprivate >> value? Anyway, if it fixes stuff it's great :) >> >> I think the patch should then also remove the firstprivate() clause, >> since that variable wouldn't be initialized. It was only there to make >> compiler warnings to away. >> >> Mark > > I opened a pull request. Let's discuss it there. > > Mark, could you comment on the pull request and/or merge it then? Stefan From markflorisson88 at gmail.com Wed Aug 29 22:49:07 2012 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 29 Aug 2012 21:49:07 +0100 Subject: [Cython] Cython 0.17 beta 3 released - release candidate In-Reply-To: <503E7973.9030803@behnel.de> References: <503672BF.5070702@behnel.de> <50370F2F.4090503@uci.edu> <5037D950.1010106@behnel.de> <50383376.7090500@uci.edu> <503AC1B3.90807@uci.edu> <503B9AD4.2000707@uci.edu> <503E7973.9030803@behnel.de> Message-ID: On 29 August 2012 21:20, Stefan Behnel wrote: > Christoph Gohlke, 27.08.2012 18:05: >> On 8/27/2012 2:42 AM, mark florisson wrote: >>> On 27 August 2012 01:39, Christoph Gohlke wrote: >>>> On my system, the following patch fixes all the lastprivate related test >>>> errors and does not have any side effects on other tests. It removes the >>>> additional initialization of the target index after `#pragma omp parallel`: >>>> >>>> diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py >>>> index 188de3d..5ba0eee 100644 >>>> --- a/Cython/Compiler/Nodes.py >>>> +++ b/Cython/Compiler/Nodes.py >>>> @@ -7627,7 +7627,7 @@ class ParallelRangeNode(ParallelStatNode): >>>> # target index uninitialized >>>> code.putln("if (%(nsteps)s > 0)" % fmt_dict) >>>> code.begin_block() # if block >>>> - code.putln("%(target)s = 0;" % fmt_dict) >>>> self.generate_loop(code, fmt_dict) >>>> code.end_block() # end if block >>>> >>>> I'm down to one remaining test failure on win32-py2.7 >>> >>> Thanks, that's useful. Could you make a pull request? Maybe because >>> the entry to the parallel loop has no barrier the assignment comes >>> after the lastprivate? I'm not sure how that's possible, I would >>> expect the lastprivate to be executed after the barrier by the highest >>> ranking thread, so there would be no race. Maybe the last thread is >>> slow, and doesn't get any iterations, and assigns the firstprivate >>> value? Anyway, if it fixes stuff it's great :) >>> >>> I think the patch should then also remove the firstprivate() clause, >>> since that variable wouldn't be initialized. It was only there to make >>> compiler warnings to away. >>> >>> Mark >> >> I opened a pull request. Let's discuss it there. >> >> > > Mark, could you comment on the pull request and/or merge it then? > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel Oh sorry, yeah it looks good, I merged it. From stefan_ml at behnel.de Thu Aug 30 11:22:19 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 30 Aug 2012 11:22:19 +0200 Subject: [Cython] Cython 0.17 beta 4 released - final release candidate Message-ID: <503F30CB.2080305@behnel.de> Hello everyone, on behalf of the Cython project team, I'm proud to announce the release of our fourth beta of Cython 0.17. Unless something truly unforseen happens, this will become our final release. So please give it another test run against your code. Cython 0.17 is another major step forward in the development of the language that will make life easier for a lot of users, rounds up some rough edges of the compiler and adds (preliminary) support for CPython 3.3 and PyPy. Download: http://cython.org/release/Cython-0.17b4.tar.gz Release notes: http://wiki.cython.org/ReleaseNotes-0.17 Documentation: http://docs.cython.org/ Major features of this release include: * vastly improved integration with the C++ STL containers http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html#standard-library http://docs.cython.org/src/tutorial/strings.html#c-strings * "yield from" delegation between generators (PEP 380) http://www.python.org/dev/peps/pep-0380/ * alpha quality support for PyPy (via cpyext) http://docs.cython.org/src/userguide/pypy.html Several other features and improvements are listed in the release notes: http://wiki.cython.org/ReleaseNotes-0.17 Have fun, Stefan From sturla at molden.no Thu Aug 30 17:42:35 2012 From: sturla at molden.no (Sturla Molden) Date: Thu, 30 Aug 2012 17:42:35 +0200 Subject: [Cython] array expressions In-Reply-To: References: Message-ID: <503F89EB.3060802@molden.no> While I have not tried this yet, array expressions in Cython might be the final nails in the coffin for Fortran 90 as far as I am concerned. Great work! :-) Sturla On 24.08.2012 20:40, mark florisson wrote: > Hey, > > Here a pull request for element-wise array expressions for Cython: > https://github.com/cython/cython/pull/144 > It includes the IndexNode refactoring branch as well. > > This has been the work this last summer for the gsoc, with great > supervision from Dag, who helped steer the project in a great > direction to make it reusable (it's partially included in Numba and > will likely be in Theano in the future, hopefully others as well). I > also wrote a thesis for my master's, which can be found here > https://github.com/markflorisson88/minivect/tree/master/thesis, which > can shed > some light on some parts of the design and performance aspects. > Performance graphs can also be found here: > https://github.com/markflorisson88/minivect/tree/master/bench/graphs > > So anyway, how would you prefer dealing with the minivect submodule? > We could include it verbatim, with any modifications made to minivect > directly, since we'd have separate git histories. We could > alternatively make it an optional submodule which is only required > when actually using array expressions. I like the latter, but anything > is fine with me really. > > Mark > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Thu Aug 30 19:54:55 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 30 Aug 2012 19:54:55 +0200 Subject: [Cython] Hashability of Python memoryviews Message-ID: <503FA8EF.4040203@behnel.de> Hi, there's a ticket on the CPython bug tracker where correct hashability of the "memoryview" type is being discussed. I don't have an opinion on it, but some of you might. It could translate into a similar feature for Cython's array and memory view types (no idea if hashability has been considered there yet). http://bugs.python.org/issue15814 Stefan From dalcinl at gmail.com Fri Aug 31 04:15:30 2012 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 30 Aug 2012 23:15:30 -0300 Subject: [Cython] Hashability of Python memoryviews In-Reply-To: <503FA8EF.4040203@behnel.de> References: <503FA8EF.4040203@behnel.de> Message-ID: On 30 August 2012 14:54, Stefan Behnel wrote: > Hi, > > there's a ticket on the CPython bug tracker where correct hashability of > the "memoryview" type is being discussed. I don't have an opinion on it, > but some of you might. It could translate into a similar feature for > Cython's array and memory view types (no idea if hashability has been > considered there yet). > readonly != immutable -> unhashable -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169