From stefan_ml at behnel.de Sat Feb 1 17:04:19 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 01 Feb 2014 17:04:19 +0100 Subject: [Cython] inspect.isbuiltin(cyfunction) and inheritance from PyCFunction (was: breaking news in Py3.4: inspect.isfunction(cyfunction) == True) In-Reply-To: <52EC012B.9040907@behnel.de> References: <52EC012B.9040907@behnel.de> Message-ID: <52ED1B03.9050109@behnel.de> Stefan Behnel, 31.01.2014 21:01: > Yury Selivanov just committed a revised version of a patch I wrote for > CPython's inspect module last year. It now accepts Cython's function type > as Python function, based on the function-like attributes that it exports. > > http://hg.python.org/cpython/rev/32a660a52aae > > That means that things like inspect.signature(cyfunction) will also work > now, and finally have a reason to be further improved, including parameter > introspection, annotations, etc. :) Sorry, huge correction: inspect.isfunction(cyfunction) still returns False. Only introspection and signature analysis in the inspect module were changed. I got it wrong because I had code in my usersitecustomize.py that monkey patches inspect.isfunction(), and had completely forgotten about it. Looking through this a bit more, I noticed that CyFunction inherits from PyCFunction, but doesn't actually tell CPython that it does. If I add this line to __Pyx_CyFunction_init(), before the PyType_Ready() call: __pyx_CyFunctionType_type.tp_base = &PyCFunction_Type; it still works as expected, but inspect.isbuiltin() now returns True (as expected). Does anyone see a reason why we should not be doing this? Stefan From stefan_ml at behnel.de Sun Feb 2 11:07:54 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Feb 2014 11:07:54 +0100 Subject: [Cython] inspect.isbuiltin(cyfunction) and inheritance from PyCFunction In-Reply-To: <52ED1B03.9050109@behnel.de> References: <52EC012B.9040907@behnel.de> <52ED1B03.9050109@behnel.de> Message-ID: <52EE18FA.3000003@behnel.de> Stefan Behnel, 01.02.2014 17:04: > Stefan Behnel, 31.01.2014 21:01: >> Yury Selivanov just committed a revised version of a patch I wrote for >> CPython's inspect module last year. It now accepts Cython's function type >> as Python function, based on the function-like attributes that it exports. >> >> http://hg.python.org/cpython/rev/32a660a52aae >> >> That means that things like inspect.signature(cyfunction) will also work >> now, and finally have a reason to be further improved, including parameter >> introspection, annotations, etc. :) > > Sorry, huge correction: inspect.isfunction(cyfunction) still returns False. > Only introspection and signature analysis in the inspect module were > changed. I got it wrong because I had code in my usersitecustomize.py that > monkey patches inspect.isfunction(), and had completely forgotten about it. > > Looking through this a bit more, I noticed that CyFunction inherits from > PyCFunction, but doesn't actually tell CPython that it does. If I add this > line to __Pyx_CyFunction_init(), before the PyType_Ready() call: > > __pyx_CyFunctionType_type.tp_base = &PyCFunction_Type; > > it still works as expected, but inspect.isbuiltin() now returns True (as > expected). Does anyone see a reason why we should not be doing this? Tried it, found a reason. It makes CPython inherit some slot functions from the base type, specifically tp_hash and tp_richompare, and those don't work with the cyfunction type. Meaning, we'd have to implement those explicitly to override the base type's implementations. Stefan From stefan_ml at behnel.de Sun Feb 2 13:12:26 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Feb 2014 13:12:26 +0100 Subject: [Cython] inspect.isbuiltin(cyfunction) and inheritance from PyCFunction In-Reply-To: <52EE18FA.3000003@behnel.de> References: <52EC012B.9040907@behnel.de> <52ED1B03.9050109@behnel.de> <52EE18FA.3000003@behnel.de> Message-ID: <52EE362A.5020803@behnel.de> Stefan Behnel, 02.02.2014 11:07: > Stefan Behnel, 01.02.2014 17:04: >> Stefan Behnel, 31.01.2014 21:01: >>> Yury Selivanov just committed a revised version of a patch I wrote for >>> CPython's inspect module last year. It now accepts Cython's function type >>> as Python function, based on the function-like attributes that it exports. >>> >>> http://hg.python.org/cpython/rev/32a660a52aae >>> >>> That means that things like inspect.signature(cyfunction) will also work >>> now, and finally have a reason to be further improved, including parameter >>> introspection, annotations, etc. :) >> >> Sorry, huge correction: inspect.isfunction(cyfunction) still returns False. >> Only introspection and signature analysis in the inspect module were >> changed. I got it wrong because I had code in my usersitecustomize.py that >> monkey patches inspect.isfunction(), and had completely forgotten about it. >> >> Looking through this a bit more, I noticed that CyFunction inherits from >> PyCFunction, but doesn't actually tell CPython that it does. If I add this >> line to __Pyx_CyFunction_init(), before the PyType_Ready() call: >> >> __pyx_CyFunctionType_type.tp_base = &PyCFunction_Type; >> >> it still works as expected, but inspect.isbuiltin() now returns True (as >> expected). Does anyone see a reason why we should not be doing this? > > Tried it, found a reason. It makes CPython inherit some slot functions from > the base type, specifically tp_hash and tp_richompare, and those don't work > with the cyfunction type. Meaning, we'd have to implement those explicitly > to override the base type's implementations. I think I figured it out. The fact that the object struct starts with "PyCFunctionObject" is more of a convenience. There is no "real" inheritance relation between the two, and that's the right thing to do, also. So, the situation in Py3.4 is as follows. inspect.isfunction(cyfunction) -> False inspect.isbuiltin(cyfunction) -> False inspect.isroutine(cyfunction) -> True inspect.signature(cyfunction) -> Signature The fact that isroutine() returns True seems like a coincidence, based on the fact that ismethoddescriptor() returns True, which sounds kind of wrong. It just tests for an existing "__get__" on the type without a matching "__set__". Anyway, I think that's pretty much as good as we can ask for, given that Cython's function type really just quacks like a Python function. Stefan From stefan_ml at behnel.de Sun Feb 2 18:13:03 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 02 Feb 2014 18:13:03 +0100 Subject: [Cython] Bug: _PyType_Lookup shortcut in Cython 0.20 breaks lookup of __exit__ in Python 2.6 In-Reply-To: <1391355615.6795.1.camel@localhost> References: <1391355615.6795.1.camel@localhost> Message-ID: <52EE7C9F.4070707@behnel.de> [forwarding to cython-devel] Wouter Bolsterlee, 02.02.2014 16:40: > Cython 0.20 introduced __Pyx_PyObject_LookupSpecial as a shortcut to > lookup special methods. The commit that introduced this change is > 8f6412275c4c2d1dcf43ad40306f858b114104ed and can be seen here: > > https://github.com/cython/cython/commit/8f6412275c4c2d1dcf43ad40306f858b114104ed > > It seems this change does not work with Python 2.6. The newly introduced > function calls _PyType_Lookup(), which fails to lookup (at least) > __exit__ (on a threading.Lock) instance when run under Python 2.6. The > exact call is here: > > https://github.com/cython/cython/commit/8f6412275c4c2d1dcf43ad40306f858b114104ed#diff-83083bdce5dcca284394ead83b8d3f99R1053 > > For my Plyvel project (https://plyvel.readthedocs.org/ and > https://github.com/wbolster/plyvel) this issue manifests as follows: > > $ cython --version > Cython version 0.20 > > $ python > Python 2.6.8 (unknown, Jan 26 2013, 14:35:25) > [GCC 4.7.2] on linux2 > Type "help", "copyright", "credits" or "license" for more > information. > >>> import plyvel > >>> db = plyvel.DB('/tmp/testdb', create_if_missing=True) > >>> db.close() > Traceback (most recent call last): > File "", line 1, in > File "_plyvel.pyx", line 246, in plyvel._plyvel.DB.close > (plyvel/_plyvel.cpp:3563) > File "_plyvel.pyx", line 252, in plyvel._plyvel.DB.close > (plyvel/_plyvel.cpp:3221) > AttributeError: __exit__ > >>> > Exception AttributeError: '__exit__' in > 'plyvel._plyvel.DB.__dealloc__' ignored > > The same Cython source code works fine against Python 2.7, 3.2, and 3.3. > Reverting to Cython 0.19 makes the code run correctly on Python 2.6, > 2.7, 3.2, and 3.3 again. This means only the combination of Python 2.6 > and Cython 0.20 fails. > > To check for yourself, you can run "make test" on a Git clone of Plyvel > with either Cython 0.19 or Cython 0.20 installed, and spot the > differences. Or run "tox" to automatically test against multiple Python > versions. Ok, I don't know what the exact difference is, but I do believe you that Py2.6 doesn't play as nicely here as Py3. In any case, the CPython code that I adapted it from didn't actually exist in Py2.6/3.1, so the guarantees don't need to be there either. Given that this is essentially an optimisation (and otherwise meant to *improve* the compatibility with CPython's own behaviour), I'll disable this in Py2.6 and Py3.1 for the next bug fix release. https://github.com/cython/cython/commit/b8e37bc15373f303691de2d256ad99045221c9e0 It would be good if you could come up with a failing regression test case for this, so that we can make sure it actually keeps working in the future. > Please CC me when responding, because I'm not subscribed to > cython-devel. cython-devel is actually fairly low traffic, so temporarily subscribing to it won't hurt much. Stefan From uws at xs4all.nl Sun Feb 2 18:43:06 2014 From: uws at xs4all.nl (Wouter Bolsterlee) Date: Sun, 02 Feb 2014 18:43:06 +0100 Subject: [Cython] Bug: _PyType_Lookup shortcut in Cython 0.20 breaks lookup of __exit__ in Python 2.6 In-Reply-To: <52EE7C9F.4070707@behnel.de> References: <1391355615.6795.1.camel@localhost> <52EE7C9F.4070707@behnel.de> Message-ID: <1391362986.23608.2.camel@localhost> Hi, Thanks for the quick follow-up. See inline comments below. Stefan Behnel schreef op zo 02-02-2014 om 18:13 [+0100]: > Ok, I don't know what the exact difference is, but I do believe you > that > Py2.6 doesn't play as nicely here as Py3. In any case, the CPython > code > that I adapted it from didn't actually exist in Py2.6/3.1, so the > guarantees don't need to be there either. I didn't really look into this, so you may be right here. > Given that this is essentially an optimisation (and otherwise meant to > *improve* the compatibility with CPython's own behaviour), I'll > disable > this in Py2.6 and Py3.1 for the next bug fix release. > > https://github.com/cython/cython/commit/b8e37bc15373f303691de2d256ad99045221c9e0 > > It would be good if you could come up with a failing regression test > case > for this, so that we can make sure it actually keeps working in the > future. I'm not familiar with the Cython codebase at all, so this might be a bit too hard for me. However, I did manage to produce a minimal example that triggers the issue at hand. I've put up the relevant files here: https://gist.github.com/wbolster/8771899 Hopefully you'll be able to construct a test from this. ? Wouter -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part URL: From stefan_ml at behnel.de Mon Feb 3 08:37:23 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 03 Feb 2014 08:37:23 +0100 Subject: [Cython] Bug: _PyType_Lookup shortcut in Cython 0.20 breaks lookup of __exit__ in Python 2.6 In-Reply-To: <1391362986.23608.2.camel@localhost> References: <1391355615.6795.1.camel@localhost> <52EE7C9F.4070707@behnel.de> <1391362986.23608.2.camel@localhost> Message-ID: <52EF4733.3080407@behnel.de> Wouter Bolsterlee, 02.02.2014 18:43: > https://gist.github.com/wbolster/8771899 > > Hopefully you'll be able to construct a test from this. Thanks! https://github.com/cython/cython/commit/e40d032f4a1e851c9bc897c36176483b6c92cce8 Stefan From robertwb at gmail.com Tue Feb 4 08:33:03 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 3 Feb 2014 23:33:03 -0800 Subject: [Cython] 0.20.1 bug fix release (was: Broken list multiplication) In-Reply-To: <52EB8DB1.1070708@behnel.de> References: <52E93454.9030707@behnel.de> <52EA8070.7090108@behnel.de> <52EB8DB1.1070708@behnel.de> Message-ID: On Fri, Jan 31, 2014 at 3:49 AM, Stefan Behnel wrote: > Stefan Behnel, 30.01.2014 17:40: >> Robert Bradshaw, 29.01.2014 19:10: >>> Ouch. Between this and the other Sage issue, I feel a small point >>> release coming up. Any other bug fixes we should get in? >> >> I have a couple of pending requests for others to test lxml on Windows and >> maybe even PyPy, so let's wait another couple of days for them to report >> back. I don't see any notable changes coming. > > All confirmed that it works now, so I'm fine with getting the bug fix > release rolling. I've been looking through the changes, and most seem fairly safe (famous last words...). Is there anything in master that shouldn't go in? > The Sage builds looks a bit broken currently, though. Yeah, I've been trying to figure out what's up with that... - Robert From stefan_ml at behnel.de Tue Feb 4 11:07:15 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 04 Feb 2014 11:07:15 +0100 Subject: [Cython] 0.20.1 bug fix release In-Reply-To: References: <52E93454.9030707@behnel.de> <52EA8070.7090108@behnel.de> <52EB8DB1.1070708@behnel.de> Message-ID: <52F0BBD3.2050309@behnel.de> Robert Bradshaw, 04.02.2014 08:33: > On Fri, Jan 31, 2014 at 3:49 AM, Stefan Behnel wrote: >> Stefan Behnel, 30.01.2014 17:40: >>> Robert Bradshaw, 29.01.2014 19:10: >>>> Ouch. Between this and the other Sage issue, I feel a small point >>>> release coming up. Any other bug fixes we should get in? >>> >>> I have a couple of pending requests for others to test lxml on Windows and >>> maybe even PyPy, so let's wait another couple of days for them to report >>> back. I don't see any notable changes coming. >> >> All confirmed that it works now, so I'm fine with getting the bug fix >> release rolling. > > I've been looking through the changes, and most seem fairly safe I guess you saw the CyFunction struct layout change? Given that the type sharing is done at a per-version level, even that should be safe. > Is there anything in master that shouldn't go in? Well, given that there currently isn't a CPython 3.4 pre-release that actually triggers the tests I added, it's a bit of a bet to keep them in. I tried them on my local checkout, but there is still some churn and discussion going on in CPython, which means that some things can still change. OTOH, the worst thing that can happen is that we end up with a point release where some tests will fail in a future Py3.4 release, and that we'd need yet another point release for the final Py3.4 support (which we may need anyway, because it's a moving target as of now). So, I don't think releasing master as it is really hurts anyone. Stefan From stefan_ml at behnel.de Wed Feb 5 18:00:15 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Feb 2014 18:00:15 +0100 Subject: [Cython] CPython ticket on allowing non-ASCII names for extension modules Message-ID: <52F26E1F.8070609@behnel.de> There's a ticket about allowing extension modules to have non-ASCII names. http://bugs.python.org/issue20485 Stefan From joshua.adelman at gmail.com Thu Feb 6 16:21:04 2014 From: joshua.adelman at gmail.com (Joshua Adelman) Date: Thu, 6 Feb 2014 10:21:04 -0500 Subject: [Cython] Bug: Memoryview of struct with adjacent string fields does not detect string boundaries Message-ID: This discussion was initially started on the cython user google group, but I wanted to move the issue over to the dev list, as per the suggestion on cython_trac. Given a numpy recarray containing two or more fixed-length string fields, if those string fields are adjacent to one another cython does not properly detect the boundary between the string fields. A concise test case demonstrating the problem is: ```cython cimport numpy as np cdef packed struct tstruct: np.float32_t a np.int16_t b char[6] c char[4] d def test_struct(tstruct[:] x): pass ``` We then define some data on the python side: ```python import numpy as np a = np.recarray(3, dtype=[('a', np.float32), ('b', np.int16), ('c', '|S6'), ('d', '|S4')]) a[0] = (1.1, 1, 'abcde', 'fgh') a[1] = (2.1, 2, 'ijklm', 'nop') a[2] = (3.1, 3, 'qrstu', 'vwx') test_struct(a) ``` This results in the error: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () ----> 1 test_struct(a) ValueError: Expected a dimension of size 6, got 10 If we swap the order of the fields in the recarray and `tstruct` to (a,c,b,d) so that there is a numerical field between the string fields, then the function can parse the memory view correctly. The relevant line of code that catches the incorrect value of `enc_count` is: https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L468 ``` if (ctx->enc_count != ctx->head->field->type->arraysize[0]) { PyErr_Format(PyExc_ValueError, "Expected a dimension of size %zu, got %zu", ctx->head->field->type->arraysize[0], ctx->enc_count); return -1; } ``` My naive guess is that there is something going on in: https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L738 since that appears to be the only place where `enc_count` is being incremented. That would seem like the place where a boundary between two string fields might not be properly handled (the comment in the line above "Continue pooling same type" is suggestive. I'll cross-post this on the cython trac once I have access and will then submit a pull request on Github of a test case once I have the trac issue number. Josh From robertwb at gmail.com Sat Feb 8 07:13:18 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 7 Feb 2014 22:13:18 -0800 Subject: [Cython] Bugfix release 0.20.1 release candidate Message-ID: Tarball up at http://cython.org/release/Cython-0.20.1rc1.tar.gz This is a bugfix only release, primarily to address the sequence literal multiplication bug, but the set of improvements since the last release looked relatively safe so I've pulled in everything from master (if that turns out to be a problem we'll release with just this big fix). Let us know if it breaks anything; I hope to get this out soon. - Robert From stefan_ml at behnel.de Sat Feb 8 19:58:31 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Feb 2014 19:58:31 +0100 Subject: [Cython] Bug: Memoryview of struct with adjacent string fields does not detect string boundaries In-Reply-To: References: Message-ID: <52F67E57.2040003@behnel.de> Joshua Adelman, 06.02.2014 16:21: > This discussion was initially started on the cython user google group, but I wanted to move the issue over to the dev list, as per the suggestion on cython_trac. Original discussion is here: http://thread.gmane.org/gmane.comp.python.cython.user/10646 > Given a numpy recarray containing two or more fixed-length string fields, if those string fields are adjacent to one another cython does not properly detect the boundary between the string fields. A concise test case demonstrating the problem is: > > ```cython > cimport numpy as np > > cdef packed struct tstruct: > np.float32_t a > np.int16_t b > char[6] c > char[4] d > > def test_struct(tstruct[:] x): > pass > ``` > > We then define some data on the python side: > > ```python > import numpy as np > > a = np.recarray(3, dtype=[('a', np.float32), ('b', np.int16), ('c', '|S6'), ('d', '|S4')]) > a[0] = (1.1, 1, 'abcde', 'fgh') > a[1] = (2.1, 2, 'ijklm', 'nop') > a[2] = (3.1, 3, 'qrstu', 'vwx') > > test_struct(a) > ``` > > This results in the error: > > --------------------------------------------------------------------------- > ValueError Traceback (most recent call last) > > in () > ----> 1 test_struct(a) > > ValueError: Expected a dimension of size 6, got 10 > > > If we swap the order of the fields in the recarray and `tstruct` to (a,c,b,d) so that there is a numerical field between the string fields, then the function can parse the memory view correctly. > > The relevant line of code that catches the incorrect value of `enc_count` is: > https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L468 > > ``` > if (ctx->enc_count != ctx->head->field->type->arraysize[0]) { > PyErr_Format(PyExc_ValueError, > "Expected a dimension of size %zu, got %zu", > ctx->head->field->type->arraysize[0], ctx->enc_count); > return -1; > } > ``` > > My naive guess is that there is something going on in: > https://github.com/cython/cython/blob/master/Cython/Utility/Buffer.c#L738 i.e. this code: """ case 'O': case 's': case 'p': if (ctx->enc_type == *ts && got_Z == ctx->is_complex && ctx->enc_packmode == ctx->new_packmode) { /* Continue pooling same type */ ctx->enc_count += ctx->new_count; } else { /* New type */ if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL; ctx->enc_count = ctx->new_count; ctx->enc_packmode = ctx->new_packmode; ctx->enc_type = *ts; ctx->is_complex = got_Z; } ++ts; ctx->new_count = 1; got_Z = 0; break; """ > since that appears to be the only place where `enc_count` is being incremented. That would seem like the place where a boundary between two string fields might not be properly handled (the comment in the line above "Continue pooling same type" is suggestive. Yes. That code is basically unchanged since Dag committed it back in 2009 (CC-ing him). However, this parser is fairly complex (not surprisingly, the buffer type format *is* complicated), so I'm not sure about the right fix. My guess is that the first part of the conditional belongs more in the "case 'Z'" fall-through section that precedes the code above. Hard to say. Even when I disable it ("if (0 &&...)"), I don't get any test failures in the format tests. In any case, I agree with you that the "pooling same size" is the bit that goes wrong here. I've attached a test case for now. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: buffer_format_test.patch Type: text/x-patch Size: 665 bytes Desc: not available URL: From robertwb at gmail.com Wed Feb 12 06:35:18 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 11 Feb 2014 21:35:18 -0800 Subject: [Cython] Cython 0.20.1 released Message-ID: I just pushed the 0.20.1 bugfix release. The changelist can be found at https://github.com/cython/cython/blob/0.20.x/CHANGES.rst , but most notable is the first entry which corrects a bug optimizing literal sequence multiplication. - Robert From arfrever.fta at gmail.com Wed Feb 12 22:01:05 2014 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Wed, 12 Feb 2014 22:01:05 +0100 Subject: [Cython] Cython 0.20.1 released In-Reply-To: References: Message-ID: <201402122201.06361.Arfrever.FTA@gmail.com> Error occurs in Cython.Debugger.Tests.TestLibCython.TestAll.test_all(): ====================================================================== ERROR: test_all (Cython.Debugger.Tests.TestLibCython.TestAll) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/Cython-0.20.1/Cython/Debugger/Tests/TestLibCython.py", line 188, in setUp super(GdbDebuggerTestCase, self).setUp() File "/tmp/Cython-0.20.1/Cython/Debugger/Tests/TestLibCython.py", line 126, in setUp **opts TypeError: __init__() takes at least 5 non-keyword arguments (4 given) ---------------------------------------------------------------------- Ran 8372 tests in 3569.636s FAILED (errors=1) ALL DONE -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From arfrever.fta at gmail.com Fri Feb 14 09:34:20 2014 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Fri, 14 Feb 2014 09:34:20 +0100 Subject: [Cython] Cython 0.20.1 released In-Reply-To: <201402122201.06361.Arfrever.FTA@gmail.com> References: <201402122201.06361.Arfrever.FTA@gmail.com> Message-ID: <201402140934.23093.Arfrever.FTA@gmail.com> 2014-02-12 22:01 Arfrever Frehtes Taifersar Arahesis napisa?(a): > Error occurs in Cython.Debugger.Tests.TestLibCython.TestAll.test_all(): > > > ====================================================================== > ERROR: test_all (Cython.Debugger.Tests.TestLibCython.TestAll) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/tmp/Cython-0.20.1/Cython/Debugger/Tests/TestLibCython.py", line 188, in setUp > super(GdbDebuggerTestCase, self).setUp() > File "/tmp/Cython-0.20.1/Cython/Debugger/Tests/TestLibCython.py", line 126, in setUp > **opts > TypeError: __init__() takes at least 5 non-keyword arguments (4 given) > > ---------------------------------------------------------------------- > Ran 8372 tests in 3569.636s > > FAILED (errors=1) > ALL DONE This problem was probably introduced in commit cdf5f0de5a0fd3d99857fd33468f24952248607f, which changed signature of runtests.CythonCompileTestCase.__init__(), but failed to update call to this function in Cython/Debugger/Tests/TestLibCython.py. https://github.com/cython/cython/commit/cdf5f0de5a0fd3d99857fd33468f24952248607f -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From stefan_ml at behnel.de Fri Feb 14 09:57:44 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 14 Feb 2014 09:57:44 +0100 Subject: [Cython] Cython 0.20.1 released In-Reply-To: <201402140934.23093.Arfrever.FTA@gmail.com> References: <201402122201.06361.Arfrever.FTA@gmail.com> <201402140934.23093.Arfrever.FTA@gmail.com> Message-ID: <52FDDA88.4080008@behnel.de> Arfrever Frehtes Taifersar Arahesis, 14.02.2014 09:34: > 2014-02-12 22:01 Arfrever Frehtes Taifersar Arahesis napisa?(a): >> Error occurs in Cython.Debugger.Tests.TestLibCython.TestAll.test_all(): >> >> ====================================================================== >> ERROR: test_all (Cython.Debugger.Tests.TestLibCython.TestAll) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/tmp/Cython-0.20.1/Cython/Debugger/Tests/TestLibCython.py", line 188, in setUp >> super(GdbDebuggerTestCase, self).setUp() >> File "/tmp/Cython-0.20.1/Cython/Debugger/Tests/TestLibCython.py", line 126, in setUp >> **opts >> TypeError: __init__() takes at least 5 non-keyword arguments (4 given) >> >> ---------------------------------------------------------------------- >> Ran 8372 tests in 3569.636s >> >> FAILED (errors=1) >> ALL DONE > > This problem was probably introduced in commit cdf5f0de5a0fd3d99857fd33468f24952248607f, which > changed signature of runtests.CythonCompileTestCase.__init__(), but failed to update call to > this function in Cython/Debugger/Tests/TestLibCython.py. > > https://github.com/cython/cython/commit/cdf5f0de5a0fd3d99857fd33468f24952248607f Thanks! I pushed a fix to master. Stefan From gael.varoquaux at normalesup.org Mon Feb 17 12:46:06 2014 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 17 Feb 2014 12:46:06 +0100 Subject: [Cython] Invalid C++ with Python and C++ iterators (not compiling CLang) Message-ID: <20140217114606.GA2787@phare.normalesup.org> Hi, I'd like to enquire on the status of the problem raised last year: https://mail.python.org/pipermail/cython-devel/2013-April/003629.html To summarize, the following Cython leads to code that doesn't compile with Clang and is probably invalid C++: from libcpp.vector cimport vector as libcpp_vector from cython.operator cimport dereference as deref, preincrement as inc cdef class TestClass: cdef libcpp_vector[float] inst def __iter__(self): it = self.inst.begin() while it != self.inst.end(): yield deref(it) inc(it) This create the following C++ code: p->__pyx_v_it.std::vector::iterator::~iterator(); which is invalid, but happens to work on GCC and MSVC. However, Clang support is starting to become very important, as it is the default compiler on MacOSX. The original reporter suggested a valid C++ code (see original email), which wasn't very convenient as it required a typedef. Any progress on the issue or suggestion to work around? Cheers, Ga?l From robertwb at gmail.com Fri Feb 21 22:12:13 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 21 Feb 2014 13:12:13 -0800 Subject: [Cython] Invalid C++ with Python and C++ iterators (not compiling CLang) In-Reply-To: <20140217114606.GA2787@phare.normalesup.org> References: <20140217114606.GA2787@phare.normalesup.org> Message-ID: For reference, this has been fixed: https://groups.google.com/forum/#!topic/cython-users/2GO2_JRXr9s On Mon, Feb 17, 2014 at 3:46 AM, Gael Varoquaux wrote: > Hi, > > I'd like to enquire on the status of the problem raised last year: > https://mail.python.org/pipermail/cython-devel/2013-April/003629.html > > To summarize, the following Cython leads to code that doesn't compile > with Clang and is probably invalid C++: > > from libcpp.vector cimport vector as libcpp_vector > from cython.operator cimport dereference as deref, preincrement as inc > > cdef class TestClass: > > cdef libcpp_vector[float] inst > > def __iter__(self): > it = self.inst.begin() > while it != self.inst.end(): > yield deref(it) > inc(it) > > This create the following C++ code: > > p->__pyx_v_it.std::vector::iterator::~iterator(); > > which is invalid, but happens to work on GCC and MSVC. However, Clang > support is starting to become very important, as it is the default > compiler on MacOSX. > > The original reporter suggested a valid C++ code (see original email), > which wasn't very convenient as it required a typedef. > > Any progress on the issue or suggestion to work around? > > Cheers, > > Ga?l > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Sat Feb 22 10:02:26 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 22 Feb 2014 10:02:26 +0100 Subject: [Cython] 0.20.2 Message-ID: <530867A2.4090604@behnel.de> Hi, there are a couple of nice additions and fixes in the current master that could go out as a 0.20.2 some time soon. I suggest waiting another couple of days for more bug reports to come in and then just send it out. I'd like to also fix the buffer formatting bug by then, looks like I got a hold of it. Stefan From andriy.kornatskyy at live.com Sat Feb 22 10:35:58 2014 From: andriy.kornatskyy at live.com (Andriy Kornatskyy) Date: Sat, 22 Feb 2014 11:35:58 +0200 Subject: [Cython] Compiler crash in RemoveUnreachableCode In-Reply-To: <52DE2CEE.7030906@behnel.de> References: <52DD84BB.1080100@behnel.de> <52DE2CEE.7030906@behnel.de> Message-ID: Stefan, Any update to the issue reported in this message thread? If you can provide a link to the issue in cython bug tracker, so I can see progress, I would greatly appreciate that. Thanks. Andriy Kornatskyy On Jan 21, 2014, at 10:16 AM, Stefan Behnel wrote: > Andriy Kornatskyy, 20.01.2014 22:16: >> May be that issue is namespace package related? Both (wheezy.http and dependent wheezy.core) use namespace_packages directive in setuptools. > > No, I think it's a problem with importing in Cython compiled modules. ISTM > that some extension modules got "reloaded", i.e. their module dict got > cleared and they got reinitialised, thus recreating all global objects that > other modules had already imported (and still keep a reference to). > > >> Here is another bug (details below): >> >> 1. virtualenv env >> 2. env/bin/easy_install cython >> 3. env/bin/easy_install lxml wheezy.core >> >> It seems to have an issue while trying to install 2 or more libs at once. >> ... >> Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg >> Processing dependencies for lxml >> Finished processing dependencies for lxml >> Searching for wheezy.core >> Reading https://pypi.python.org/simple/wheezy.core/ >> Best match: wheezy.core 0.1.129 >> Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.tar.gz#md5=ea3d5f744bc0525d61f9fb48d897972d >> Processing wheezy.core-0.1.129.tar.gz >> Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg >> Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1 >> Traceback (most recent call last): >> ... >> File ?env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation >> self.generate_c_code(env, options, result) >> File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code >> rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback) >> File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697) >> >> File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__ >> stream = StringIO() >> TypeError: 'NoneType' object is not callable > > Looks like the same thing. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Sat Feb 22 20:40:47 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 22 Feb 2014 20:40:47 +0100 Subject: [Cython] Compiler crash in RemoveUnreachableCode In-Reply-To: References: <52DD84BB.1080100@behnel.de> <52DE2CEE.7030906@behnel.de> Message-ID: <5308FD3F.4070501@behnel.de> Andriy Kornatskyy, 22.02.2014 10:35: > On Jan 21, 2014, at 10:16 AM, Stefan Behnel wrote: >> I think it's a problem with importing in Cython compiled modules. ISTM >> that some extension modules got "reloaded", i.e. their module dict got >> cleared and they got reinitialised, thus recreating all global objects that >> other modules had already imported (and still keep a reference to). >> >>> Here is another bug (details below): >>> >>> 1. virtualenv env >>> 2. env/bin/easy_install cython >>> 3. env/bin/easy_install lxml wheezy.core >>> >>> It seems to have an issue while trying to install 2 or more libs at once. >>> ... >>> Installed env/lib/python2.7/site-packages/lxml-3.3.0beta5-py2.7-macosx-10.9-x86_64.egg >>> Processing dependencies for lxml >>> Finished processing dependencies for lxml >>> Searching for wheezy.core >>> Reading https://pypi.python.org/simple/wheezy.core/ >>> Best match: wheezy.core 0.1.129 >>> Downloading https://pypi.python.org/packages/source/w/wheezy.core/wheezy.core-0.1.129.tar.gz#md5=ea3d5f744bc0525d61f9fb48d897972d >>> Processing wheezy.core-0.1.129.tar.gz >>> Writing /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/setup.cfg >>> Running wheezy.core-0.1.129/setup.py -q bdist_egg --dist-dir /var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-UC_pgJ/wheezy.core-0.1.129/egg-dist-tmp-9sntQ1 >>> Traceback (most recent call last): >>> ... >>> File ?env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 109, in process_implementation >>> self.generate_c_code(env, options, result) >>> File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ModuleNode.py", line 302, in generate_c_code >>> rootwriter = Code.CCodeWriter(emit_linenums=emit_linenums, c_line_in_traceback=options.c_line_in_traceback) >>> File "Code.py", line 1406, in Cython.Compiler.Code.CCodeWriter.__init__ (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Code.c:30697) >>> >>> File "env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/StringIOTree.py", line 11, in __init__ >>> stream = StringIO() >>> TypeError: 'NoneType' object is not callable >> >> Looks like the same thing. > > Any update to the issue reported in this message thread? No, not really. The current setup is just way too vast to debug. I don't even know how easy_install runs the installation of a single package, nor how it's possible that two packages interfere in such a way. I'd expect it to run subprocesses, so why any interference at all? It might be that the main process loads the setup.py scripts in some way to read the meta data, thus executes their imports, and then tries to unload the modules before it loads the next setup.py script. That would be rather bad for any extension modules imported by both. So bad that's I'd consider it a bug in setuptools. Let's assume that reloading really is what's happening here. Given that this basically doesn't work for extension modules, definitely not in Py2.x and barely in Py3.x, we might be able to apply a barrier to the module init function that prevents it from running a second time, and just return the already created module somehow. But that's blank guessing and I really can't tell for sure. It would help if someone could invest some time into properly analysing this further. Stefan From scopatz at gmail.com Thu Feb 27 01:24:13 2014 From: scopatz at gmail.com (Anthony Scopatz) Date: Wed, 26 Feb 2014 18:24:13 -0600 Subject: [Cython] ANN: XDress v0.4 Message-ID: Hello All, I am *extremely *pleased to be able to announce the version 0.4 release of xdress. This version contains much anticipated full support for Clang as a parser! This is almost entirely due to the efforts of Geoffrey Irving. Please thank him the next time you get a chance :) This release also contains a lot of other goodies that you can read about in the release notes below. Happy Generating! Anthony XDress 0.4 Release Notes XDress is a numpy-aware automatic wrapper generator for C/C++ written in pure Python. Currently, xdress may generate Python bindings (via Cython) for C++ classes, functions, and certain variable types. It also contains idiomatic wrappers for C++ standard library containers (sets, vectors, maps). In the future, other tools and bindings will be supported. The main enabling feature of xdress is a dynamic type system that was designed with the purpose of API generation in mind. Release highlights: - Clang support! All kudos to Geoffrey Irving! - NumPy dtypes may be created independently of C++ STL vectors - A complete test suite refactor - Arbitrary source code locations - Global run control files - A plethora of useful bug fixes This version of xdress is *not* 100% backwards compatible with previous versions of xdress. We apologize in the name of progress. It represents ans impressive 245 files changed, 44917 aggregate line insertions (+), and 7893 deletions (-). Please visit the website for more information: http://xdress.org/ Ask questions on the mailing list: https://groups.google.com/forum/#!forum/xdress Download the code from GitHub: http://github.com/xdress/xdress XDress is free & open source (BSD 2-clause license) and requires Python 2.7+, NumPy 1.5+, Cython 0.19+, and optionally Clang, GCC-XML, pycparser, dOxygen, or lxml. New Features Clang Support Through the herculean efforts of Geoffrey Irving xdress finally has full, first-class Clang/LLVM support! This is major advancement as it allows xdress to wrap more modern versions of C++ than GCC-XML can handle. Because of deficiencies in the existing libclang and Python bindings it was necessary for us to fork libclang for xdress in the short term. We hope to integrate these changes upstream. Clang versions 3.2 - 3.4 are supported. Independent NumPy Dtypes In previous versions of xdress, to create a dtype of type T the user needed to declare the desire for a wrapper of an STL vector of type T. These two desires have now been separated. It is now possible to create a dtype via the dtypes run control parameter. STL vectors are still wrapped via dtypes. See the dtypes module for more information. Shiny New Test Suite The xdress test suite has been completely revamped to include both unit and integration tests which are run for all available parsers. The integration tests are accomplished though two fake projects - cproj and cppproj - on which the xdress CLI is run. These tests are now fully platform independent, unlike the previous BASH-based test suite. Source Paths Source file paths are now given by either their absolute or relative path. This allows source code to be located anywhere on the user's file system and enable the wrapping of dependencies or externally supplied libraries as needed. The run control parametersourcedir has been deprecated. Global Run Control Files It is sometimes useful to be able to set system-wide run control parameters. XDress will now search the following files in order of increasing precedence. - $HOME/.xdressrc - $HOME/.xdressrc.py - $HOME/.config/xdressrc - $HOME/.config/xdressrc.py $HOME is the user's home directory. Settings in the project run control file take precedence over the values here. Major Bug Fixes - Debug file now always written when in debug mode. - STL sets of custom types now allowed. - Template parameters now allowed to be enum values. - Allow classes with no default constructor. Join in the Fun! If you are interested in using xdress on your project (and need help), contributing back to xdress, starting up a development team, or writing your own code generation plugin tool, please let us know. Participation is very welcome! Authors - Anthony Scopatz - Geoffrey Irving * - James Casbon * - Kevin Tew * - Spencer Lyon - John Wiggins - Matt McCormick - Brad Buran - Chris Harris * - Gerald Dalley * - Micky Latowicki * - Mike C. Fletcher * - Robert Schwarz * An * indicates a first time contributor. Links 1. Homepage - http://xdress.org/ 2. Mailing List - https://groups.google.com/forum/#!forum/xdress 3. GitHub Organization - https://github.com/xdress -------------- next part -------------- An HTML attachment was scrubbed... URL: From syam.gadde at duke.edu Thu Feb 27 16:22:10 2014 From: syam.gadde at duke.edu (Syam Gadde) Date: Thu, 27 Feb 2014 15:22:10 +0000 Subject: [Cython] line tracing/profiling code objects Message-ID: <530F5822.1070202@duke.edu> Hi all, I tried using line tracing in conjunction with line_profiler/kernprof to attempt to see if line-based profiling works in Cython. I think it's pretty close. But I think the problem is that line_profiler is getting a different PyCodeObject when it wraps the function and when it actually gets the trace call. It adds the initial code object to a map, and later when it gets the trace call, decides that the trace call is not something it needs to pay attention to because the new code object that it gets is not the same as the original one. The first code object is created at function declaration by __Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to a variable __pyx_k_codeobj_NNN. The second code object is created, essentially, during the first entry into the function (in __Pyx_TraceCall, via __Pyx_TraceSetupAndCall). It seems that setting __pyx_frame_code to the initial code object before calling TraceCall() would fix this. Is this easy to do? I'd do it myself, but I'd need help figuring out how to get the name of the appropriate __pyx_k_codeobj_NNN variable from within FuncDefNode.generate_function_definitions(), which calls put_trace_call(). Thanks for your help... -syam From stefan_ml at behnel.de Thu Feb 27 22:06:38 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 27 Feb 2014 22:06:38 +0100 Subject: [Cython] line tracing/profiling code objects In-Reply-To: <530F5822.1070202@duke.edu> References: <530F5822.1070202@duke.edu> Message-ID: <530FA8DE.9030200@behnel.de> Hi! Syam Gadde, 27.02.2014 16:22: > I tried using line tracing in conjunction with line_profiler/kernprof to > attempt to see if line-based profiling works in Cython. I think it's > pretty close. But I think the problem is that line_profiler is getting > a different PyCodeObject when it wraps the function and when it actually > gets the trace call. It adds the initial code object to a map, and > later when it gets the trace call, decides that the trace call is not > something it needs to pay attention to because the new code object that > it gets is not the same as the original one. > > The first code object is created at function declaration by > __Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to > a variable __pyx_k_codeobj_NNN. The second code object is created, > essentially, during the first entry into the function (in > __Pyx_TraceCall, via __Pyx_TraceSetupAndCall). It seems that setting > __pyx_frame_code to the initial code object before calling TraceCall() > would fix this. That's a part of it, yes. Here's another bit in the puzzle: https://github.com/cython/cython/pull/93 The problem is that the code objects that we currently create along the way have a fixed Python code line number (because that was the simplest way to get it working). The two changes above will get us pretty far. What remains to be done then is to enable line number calculation at runtime by emulating byte code offsets in the frame object instead of using absolute line numbers. The pull request refers to CPython's Objects/lnotab_notes.txt, which has some details on the inner workings. Properly calculating line numbers at runtime would also have other nice side effects, because it would remove the need to create multiple code objects for a single function in the first place. So it's very desirable. > Is this easy to do? I'd do it myself, but I'd need help figuring out > how to get the name of the appropriate __pyx_k_codeobj_NNN variable from > within FuncDefNode.generate_function_definitions(), which calls > put_trace_call(). The best way is usually to run it through a debugger and see what you can get your hands on. :) The constant names (__pyx_k_...) are generated on the fly at C code generation time, so you can't get them before that. Two possible solutions: either change the way how the C names of code objects are being generated (e.g. by making their C name depend on the mangled C name of the function so that they can be deterministically named at analysis time), or make the code object node remember its generated constant name and reuse it in other places. As you can see, it's not entirely trivial overall, but we've already been piling up the bits and pieces so that the only remaining effort is to put everything together. If you could give it a try, I'm sure you'd make a lot of people happy with it. Stefan