From stefan_ml at behnel.de Sat Jun 6 09:54:49 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Jun 2015 09:54:49 +0200 Subject: [Cython] candidate for bug fix release 0.22.1 Message-ID: <5572A749.8090700@behnel.de> Hi everyone, here is a release candidate for the upcoming bug fix release 0.22.1. http://cython.org/release/Cython-0.22.1rc1.tar.gz http://cython.org/release/Cython-0.22.1rc1.zip cdf6726d3717351b7d4845f7b3853c3f Cython-0.22.1rc1.tar.gz e480ddd0f45289e22ea624f530116719 Cython-0.22.1rc1.zip Here's the changelog: https://github.com/cython/cython/blob/0.22.x/CHANGES.rst Please give it some testing and report success or failure, especially on less tested platforms like MS-Windows. Being a bug-fix release, it shouldn't break anything but improve the stability and compatibility of the current release series. As soon as this release is out (hopefully within a couple of days), we'll start preparing the next major feature release 0.23. Here's a teaser: https://github.com/cython/cython/blob/master/CHANGES.rst Stefan From ndbecker2 at gmail.com Sat Jun 6 20:25:48 2015 From: ndbecker2 at gmail.com (Neal Becker) Date: Sat, 06 Jun 2015 14:25:48 -0400 Subject: [Cython] cython-0.23 monkey-patches stdlib? Message-ID: According to https://github.com/cython/cython/blob/master/CHANGES.rst the default build would attempt to modify python stdlib files? How would this work on distributions? I am maintainer for Cython on Fedora. So installing Cython would try to overwrite stdlib files? That sounds difficult. For one thing, it means more than 1 package would own the same files. And I don't know if the general user community would like this approach. From stefan_ml at behnel.de Sat Jun 6 22:25:22 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 06 Jun 2015 22:25:22 +0200 Subject: [Cython] cython-0.23 monkey-patches stdlib? In-Reply-To: References: Message-ID: <55735732.9020403@behnel.de> Neal Becker schrieb am 06.06.2015 um 20:25: > According to > https://github.com/cython/cython/blob/master/CHANGES.rst > > the default build would attempt to modify python stdlib files? No. Glad you asked, that might not be clear. It will not modify the *files*, only the content of the modules after importing them. It adds the missing ABCs to the collections.abc module and registers Cython's own types with them *at runtime*. Here's what it does: https://github.com/cython/cython/blob/bb0dec2/Cython/Utility/Coroutine.c#L1322 This is a bit fragile, like all monkey-patching. As long as things get imported in the right order, it works. But if you import asyncio before a Cython module that adds the ABCs, it may already have decided that the ABCs don't exist and won't use them. That's controllable on user application side, though. Also, the first patcher necessarily wins, so if there will ever be updates to the ABCs, old Cython modules would still patch in the old versions when imported first. They should tend to be quite stable, though. In any case, this will get better over time as more code switches to Py3.5 and later. Stefan From michael at ensslin.cc Mon Jun 8 16:50:53 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Mon, 08 Jun 2015 16:50:53 +0200 Subject: [Cython] Feature request: Expose methods to fake stack traces Message-ID: <5575ABCD.2060604@ensslin.cc> Hi everybody, my C++ exceptions contain some stack trace information (starting with __FILE__ and __LINE__), and I'd like to preserve that information in my custom 'except+' translator. Cython seems to have some internal methods to fake Python stack trace objects, but research (and an unanswered question on the Users list [https://groups.google.com/forum/#!topic/cython-users/9coFCVwigpE]) seems to imply that this functionality is not currently exposed. I'd like to see that internal functionality exposed for usage from .pyx files. ~ mic_e -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From michael at ensslin.cc Thu Jun 11 18:03:32 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Thu, 11 Jun 2015 18:03:32 +0200 Subject: [Cython] Cython produces invalid code for operator() Message-ID: <5579B154.9060508@ensslin.cc> Hi guys, have a look at this: $ cat bar.pyx cdef extern from "foo.h": cdef cppclass Foo: int operator() (int arg) int do_call (int arg) cdef int bar(int arg): cdef Foo foo foo.do_call(arg) return foo(arg) $ cython3 --cplus bar.pyx $ cat bar.cpp (...) static int __pyx_f_3bar_bar(int __pyx_v_arg) { Foo __pyx_v_foo; int __pyx_r; __Pyx_RefNannyDeclarations __Pyx_RefNannySetupContext("bar", 0); /* "bar.pyx":9 * cdef int bar(int arg): * cdef Foo foo * foo.do_call(arg) # <<<<<<<<<<<<<< * return foo(arg) */ __pyx_v_foo.do_call(__pyx_v_arg); /* "bar.pyx":10 * cdef Foo foo * foo.do_call(arg) * return foo(arg) # <<<<<<<<<<<<<< */ __pyx_r = operator()(__pyx_v_arg); goto __pyx_L0; /* "bar.pyx":7 * * * cdef int bar(int arg): # <<<<<<<<<<<<<< * cdef Foo foo * foo.do_call(arg) */ /* function exit code */ __pyx_L0:; __Pyx_RefNannyFinishContext(); return __pyx_r; } (...) Note how the function invocation for "do_call" is generated correctly, but the invocation of operator() is nonsensical. The correct line would be this: __pyx_r = __pyx_v_foo(__pyx_v_arg); Happy debugging :D ~ mic_e -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From insertinterestingnamehere at gmail.com Thu Jun 11 20:19:41 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Thu, 11 Jun 2015 18:19:41 +0000 Subject: [Cython] Cython produces invalid code for operator() In-Reply-To: <5579B154.9060508@ensslin.cc> References: <5579B154.9060508@ensslin.cc> Message-ID: On Thu, Jun 11, 2015 at 10:09 AM Michael En?lin wrote: > Hi guys, > > have a look at this: > > $ cat bar.pyx > cdef extern from "foo.h": > cdef cppclass Foo: > int operator() (int arg) > int do_call (int arg) > > > cdef int bar(int arg): > cdef Foo foo > foo.do_call(arg) > return foo(arg) > > $ cython3 --cplus bar.pyx > $ cat bar.cpp > > (...) > > static int __pyx_f_3bar_bar(int __pyx_v_arg) { > Foo __pyx_v_foo; > int __pyx_r; > __Pyx_RefNannyDeclarations > __Pyx_RefNannySetupContext("bar", 0); > > /* "bar.pyx":9 > * cdef int bar(int arg): > * cdef Foo foo > * foo.do_call(arg) # <<<<<<<<<<<<<< > * return foo(arg) > */ > __pyx_v_foo.do_call(__pyx_v_arg); > > /* "bar.pyx":10 > * cdef Foo foo > * foo.do_call(arg) > * return foo(arg) # <<<<<<<<<<<<<< > */ > __pyx_r = operator()(__pyx_v_arg); > goto __pyx_L0; > > /* "bar.pyx":7 > * > * > * cdef int bar(int arg): # <<<<<<<<<<<<<< > * cdef Foo foo > * foo.do_call(arg) > */ > > /* function exit code */ > __pyx_L0:; > __Pyx_RefNannyFinishContext(); > return __pyx_r; > } > > (...) > > > > Note how the function invocation for "do_call" is generated correctly, > but the invocation of operator() is nonsensical. > > The correct line would be this: > > __pyx_r = __pyx_v_foo(__pyx_v_arg); > > Happy debugging :D > > ~ mic_e > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel There was a patch for this in https://mail.python.org/pipermail/cython-devel/2015-February/004328.html. I'm not sure what ever became of it though. I wrote up a workaround on SO at http://stackoverflow.com/a/25955546/1935144 a while ago. Best of luck! -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael at ensslin.cc Fri Jun 12 12:51:00 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Fri, 12 Jun 2015 12:51:00 +0200 Subject: [Cython] Compiler crash in AnalyseExpressionsTransform Message-ID: <557AB994.2000706@ensslin.cc> Hi, another bug report: mic at mic /tmp $ cat t11.pyx cdef cppclass foo: pass def test(): foo() mic at mic /tmp $ cython --cplus t11.pyx Error compiling Cython file: ------------------------------------------------------------ ... cdef cppclass foo: pass def test(): foo() ^ ------------------------------------------------------------ t11.pyx:5:7: Compiler crash in AnalyseExpressionsTransform ModuleNode.body = StatListNode(t11.pyx:1:0) StatListNode.stats[1] = DefNode(t11.pyx:4:0, modifiers = [...]/0, name = u'test', py_wrapper_required = True, reqd_kw_flags_cname = '0', used = True) File 'Nodes.py', line 421, in analyse_expressions: StatListNode(t11.pyx:5:7) File 'Nodes.py', line 4652, in analyse_expressions: ExprStatNode(t11.pyx:5:7) File 'ExprNodes.py', line 434, in analyse_expressions: SimpleCallNode(t11.pyx:5:7, use_managed_ref = True) File 'ExprNodes.py', line 4495, in analyse_types: SimpleCallNode(t11.pyx:5:7, use_managed_ref = True) File 'ExprNodes.py', line 4429, in analyse_as_type_constructor: SimpleCallNode(t11.pyx:5:7, use_managed_ref = True) Compiler crash traceback from this point on: File "/usr/lib/python2.7/dist-packages/Cython/Compiler/ExprNodes.py", line 4429, in analyse_as_type_constructor self.function = RawCNameExprNode(self.function.pos, constructor.type) AttributeError: 'NoneType' object has no attribute 'type' The code is obviously nonsensical, but Cython should produce a proper error message instead of crashing. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From michael at ensslin.cc Fri Jun 12 13:01:42 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Fri, 12 Jun 2015 13:01:42 +0200 Subject: [Cython] Can't call functions with multi-token template arguments such as 'char *' Message-ID: <557ABC16.8010401@ensslin.cc> Hi, it seems to be impossible to use anything but a single word as a template type for functions. Classes don't suffer from this issue, as seen below. As a workaround, complex types can be ctypedef-d to a single word (also seen below). $ cat t10.pyx cdef extern from "nope.h": cdef cppclass bar[T]: void func(T arg) void foo[T](T arg) ctypedef char * charptr def test(): # works cdef bar[char *] barobj barobj.func(NULL) # works foo[int](5) # works foo[charptr](NULL) # fails foo[char *](NULL) $ cython --cplus t10.pyx Error compiling Cython file: ------------------------------------------------------------ ... # works foo[charptr](NULL) # fails foo[char *](NULL) ^ ------------------------------------------------------------ t10.pyx:27:14: Expected an identifier or literal Happy debugging, ~ mic_e -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From michael at ensslin.cc Fri Jun 12 23:24:00 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Fri, 12 Jun 2015 23:24:00 +0200 Subject: [Cython] Feature request: Flag to enable warnings when 'extern' functions are not declared 'except +' Message-ID: <557B4DF0.2080401@localhost> Hi everybody, as you surely have guessed from the amount of Bug reports I have recently submitted to this mailing list, I'm currently working on a rather large Cython project. More precisely: I'm using Cython as the glue layer between the Python and C++ components of openage; an overview and source code may be found here: https://github.com/mic-e/openage/blob/pyinterface/doc/implementation/pyinterface.md https://github.com/mic-e/openage/tree/pyinterface/cpp/pyinterface https://github.com/mic-e/openage/tree/pyinterface/openage/cppinterface In openage, almost all C++ functions that are exposed to Cython SHOULD be declared as 'except +'. Forgetting to do so can cause hard-to-debug issues, including useless Exception messages, corruption of CPython and right-out program termination, if those methods should throw a C++ exception. As a solution, I'd like to see Cython warnings if an extern cimport has no 'except' declaration. To intentionally omit the "except +", I suggest a new except declaration, "except -", or, even better, the C++11 keyword "noexcept" (since precisely those methods that are declared 'noexcept' should be cimported without 'except+'). Of course, those warnings (or errors?) should be disabled by default, and enabled by a special command-line flag. Any thoughts on this? ~ mic_e From andrew.svetlov at gmail.com Sat Jun 13 08:30:30 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Sat, 13 Jun 2015 09:30:30 +0300 Subject: [Cython] Bug: comprehensions clear current exception Message-ID: I have an issue in aiohttp library: https://github.com/KeepSafe/aiohttp/issues/410 The source of problem is: when I execute the following code body = ', '.join("'{}': {!r}".format(k, v) for k, v in self.items()) in except block Cython clears exception. Changing from comprehension to regular iteration works well: lst = [] for k, v in self._items: lst.append("'{}': {!r}".format(k, v)) body = ', '.join(lst) In pyre Python both versions don't clear current exception. I think it's a Cython bug. -- Thanks, Andrew Svetlov From lrq3000 at gmail.com Mon Jun 15 12:34:35 2015 From: lrq3000 at gmail.com (Stephen LARROQUE) Date: Mon, 15 Jun 2015 12:34:35 +0200 Subject: [Cython] Bug: Extension Type inheriting from int cause a MemoryError Message-ID: Hello, I am trying to make an extension type inheriting from int or cython.int (to do arithmetic operations in Galois Fields). However, this causes a MemoryError because it seems such extension type is not freed correctly. Other than that, it works perfectly well. Here is a test case to reproduce the bug on Python 2.7.9 Win32 (I'm running Windows 7, but it may happen on other platform, I did not test) with Cython v0.22 on Anaconda: ######################## # mode: run import sys cdef class ExtendedInt(int): pass _ERRORS = u""" MemoryError """ # Simple test case to raise a MemoryError by generating lots of ExtendedInt total_it = 1000 for i in xrange(total_it): for j in xrange(10000000): ExtendedInt(j) sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, total_it)) ######################## My current workaround as advised in the cython-users list: https://groups.google.com/forum/#!topic/cython-users/WJ5BJ5moAh4 is to inherit from standard (object) and then reimplement all int magic methods, which is cumbersome and less optimized, but at least there's no MemoryError. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lrq3000 at gmail.com Mon Jun 15 12:37:24 2015 From: lrq3000 at gmail.com (Stephen LARROQUE) Date: Mon, 15 Jun 2015 12:37:24 +0200 Subject: [Cython] Bug: Extension Type inheriting from int cause a MemoryError In-Reply-To: References: Message-ID: Addendum: I tried to define __weakref__ and __dealloc__, but this did nothing to solve the issue. 2015-06-15 12:34 GMT+02:00 Stephen LARROQUE : > Hello, > > I am trying to make an extension type inheriting from int or cython.int (to > do arithmetic operations in Galois Fields). However, this causes a > MemoryError because it seems such extension type is not freed correctly. > Other than that, it works perfectly well. > > Here is a test case to reproduce the bug on Python 2.7.9 Win32 (I'm > running Windows 7, but it may happen on other platform, I did not test) > with Cython v0.22 on Anaconda: > > ######################## > # mode: run > > import sys > > cdef class ExtendedInt(int): pass > > _ERRORS = u""" > MemoryError > """ > > # Simple test case to raise a MemoryError by generating lots of ExtendedInt > total_it = 1000 > for i in xrange(total_it): > for j in xrange(10000000): > ExtendedInt(j) > sys.stdout.write("\rGenerating lists of ExtendedInt : %i/%i" % (i, > total_it)) > ######################## > > My current workaround as advised in the cython-users list: > https://groups.google.com/forum/#!topic/cython-users/WJ5BJ5moAh4 > > is to inherit from standard (object) and then reimplement all int magic > methods, which is cumbersome and less optimized, but at least there's no > MemoryError. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Jun 16 17:53:43 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 16 Jun 2015 17:53:43 +0200 Subject: [Cython] Bug: comprehensions clear current exception In-Reply-To: References: Message-ID: <55804687.3080806@behnel.de> Andrew Svetlov schrieb am 13.06.2015 um 08:30: > I have an issue in aiohttp library: > https://github.com/KeepSafe/aiohttp/issues/410 > > The source of problem is: when I execute the following code > > body = ', '.join("'{}': {!r}".format(k, v) for k, v in self.items()) > > in except block Cython clears exception. > > Changing from comprehension to regular iteration works well: > > lst = [] > for k, v in self._items: > lst.append("'{}': {!r}".format(k, v)) > body = ', '.join(lst) > > In pyre Python both versions don't clear current exception. Thanks for the report, I can reproduce it with this code: ''' def genexpr_in_except(x, result): """ >>> result = [] >>> try: genexpr_in_except([1, 2, 3], result) ... except ValueError: print("OK") ... else: print("NOT RAISED!") OK >>> result [2, 4, 6] >>> result = [] >>> genexpr_in_except([], result) >>> result [] """ try: if x: raise ValueError() except ValueError: assert sys.exc_info()[0] is ValueError, 1 result[:] = list(i*2 for i in x) assert sys.exc_info()[0] is ValueError, 2 raise ''' The "raise" at the end actually works, but the second "assert" makes it fail. Stefan From andrew.svetlov at gmail.com Tue Jun 16 18:31:23 2015 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Tue, 16 Jun 2015 19:31:23 +0300 Subject: [Cython] Bug: comprehensions clear current exception In-Reply-To: <55804687.3080806@behnel.de> References: <55804687.3080806@behnel.de> Message-ID: I saw the same behavior for eating current exception not only for comprehensions but for iterating over generator with yield's (both iterating code and generator was in Cython). On Tue, Jun 16, 2015 at 6:53 PM, Stefan Behnel wrote: > Andrew Svetlov schrieb am 13.06.2015 um 08:30: >> I have an issue in aiohttp library: >> https://github.com/KeepSafe/aiohttp/issues/410 >> >> The source of problem is: when I execute the following code >> >> body = ', '.join("'{}': {!r}".format(k, v) for k, v in self.items()) >> >> in except block Cython clears exception. >> >> Changing from comprehension to regular iteration works well: >> >> lst = [] >> for k, v in self._items: >> lst.append("'{}': {!r}".format(k, v)) >> body = ', '.join(lst) >> >> In pyre Python both versions don't clear current exception. > > Thanks for the report, I can reproduce it with this code: > > ''' > def genexpr_in_except(x, result): > """ > >>> result = [] > >>> try: genexpr_in_except([1, 2, 3], result) > ... except ValueError: print("OK") > ... else: print("NOT RAISED!") > OK > >>> result > [2, 4, 6] > > >>> result = [] > >>> genexpr_in_except([], result) > >>> result > [] > """ > try: > if x: > raise ValueError() > except ValueError: > assert sys.exc_info()[0] is ValueError, 1 > result[:] = list(i*2 for i in x) > assert sys.exc_info()[0] is ValueError, 2 > raise > ''' > > The "raise" at the end actually works, but the second "assert" makes it fail. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel -- Thanks, Andrew Svetlov From stefan_ml at behnel.de Fri Jun 19 08:38:32 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 19 Jun 2015 08:38:32 +0200 Subject: [Cython] Bug: Extension Type inheriting from int cause a MemoryError In-Reply-To: References: Message-ID: <5583B8E8.9050902@behnel.de> Stephen LARROQUE schrieb am 15.06.2015 um 12:34: > I am trying to make an extension type inheriting from int or cython.int (to > do arithmetic operations in Galois Fields). However, this causes a > MemoryError because it seems such extension type is not freed correctly. > Other than that, it works perfectly well. > > cdef class ExtendedInt(int): pass > > for j in xrange(10000000): > ExtendedInt(j) It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this code in intobject.c: """ static void int_dealloc(PyIntObject *v) { if (PyInt_CheckExact(v)) { Py_TYPE(v) = (struct _typeobject *)free_list; free_list = v; } else Py_TYPE(v)->tp_free((PyObject *)v); } static void int_free(PyIntObject *v) { Py_TYPE(v) = (struct _typeobject *)free_list; free_list = v; } """ Your extension type automatically inherits the "int_free" slot function from its base type, so the "else" case in "int_dealloc()" will in fact call "int_free()" and append the object to the free list despite *not* being exactly of type PyInt. Then, when creating a new ExtendedInt instance, Python's int-subtype instantiation code *ignores* the free list and instead creates a completely new object. Thus, the free list keeps growing until it fills all memory and the process dies. I created a CPython ticket. https://bugs.python.org/issue24469 I guess we could hack up a work around for this in Cython somehow (Python 2.7 is an easy target being a dead end, after all), but let's wait what the CPython devs have to say about this. Note, however, that any fix in a future CPython 2.7.x release will not fix it in earlier Python 2.x versions, so my guess is that we'd end up having to add a work-around on our side anyway. Stefan From stefan_ml at behnel.de Sat Jun 20 12:15:39 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 20 Jun 2015 12:15:39 +0200 Subject: [Cython] Cython 0.22.1 released Message-ID: <55853D4B.6080001@behnel.de> Hi all, this is to inform you that the major bug fix release 0.22.1 for the 0.22 release series has finally been published. There are no code changes from the release candidate. The full change log follows. Downloads: https://pypi.python.org/pypi/Cython/0.22.1 http://cython.org/release/Cython-0.22.1.tar.gz http://cython.org/release/Cython-0.22.1.zip Changelog: https://github.com/cython/cython/blob/146fd20d5b3603995dbee6ed77f1919f410cd010/CHANGES.rst Documentation: http://docs.cython.org/ Have fun, Stefan 0.22.1 (2015-06-20) =================== Bugs fixed ---------- * Crash when returning values on generator termination. * In some cases, exceptions raised during internal isinstance() checks were not propagated. * Runtime reported file paths of source files (e.g for profiling and tracing) are now relative to the build root directory instead of the main source file. * Tracing exception handling code could enter the trace function with an active exception set. * The internal generator function type was not shared across modules. * Comparisons of (inferred) ctuples failed to compile. * Closures inside of cdef functions returning ``void`` failed to compile. * Using ``const`` C++ references in intermediate parts of longer expressions could fail to compile. * C++ exception declarations with mapping functions could fail to compile when pre-declared in .pxd files. * C++ compilation could fail with an ambiguity error in recent MacOS-X Xcode versions. * C compilation could fail in pypy3. * Fixed a memory leak in the compiler when compiling multiple modules. * When compiling multiple modules, external library dependencies could leak into later compiler runs. Fix by Jeroen Demeyer. This fixes ticket 845. From michael at ensslin.cc Mon Jun 22 04:04:33 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Mon, 22 Jun 2015 04:04:33 +0200 Subject: [Cython] Feature Request: Variadic Templates Message-ID: <55876D31.1080905@ensslin.cc> Hi, I'd like to see support for variadic template arguments, such as this: test.h: template RetT foo(VARARGS ...); test.pyx: cdef extern from "test.h": cdef RetT foo[RetT, ... VARARGS](... VARARGS) def test(): cdef int i = foo[int, float, int](1, 2.5, 3) This would allow Cython's libcpp to easily support many of C++11's new types, including std::tuple and std::function. Support for the latter in particular would prove quite useful in passing around function pointers between Cython and C++. I have tried to implement this feature myself, but I'm entirely unfamiliar with Cython's codebase, and all my attempts ended in unsatisfactorily hacky, unstable code. I believe that any experienced Cython developer on this list would be able to properly implement this in a matter of a few hours, as it seems like a rather minor feature. The current workaround for the non-existance of this feature involves typedefs for every possible number of arguments, like this: template struct S {}; using S0 = S<>; template using S1 = S; template using S2 = S; template using S3 = S; then exporting all of S0, S1, ... individually in the .pxd file. This is inconvenient, but acceptable. Now assume that S has a member function f, template void f(Ts..., Us ...); due to the nature of C++, the same trick as above does not work, and the user will be forced to clutter the C++ code with a function body for every possible len(Us). Even worse, the .pxd file will now contain a quadratic number of wrappers (one for every possible combination of len(Ts), len(Us)), all of them extremely prone to mistakes. Thanks for reading, ~ Michael -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From insertinterestingnamehere at gmail.com Mon Jun 22 23:13:19 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Mon, 22 Jun 2015 21:13:19 +0000 Subject: [Cython] operator() bug in cython In-Reply-To: References: Message-ID: On Tue, Feb 17, 2015 at 2:51 AM Ulrich Dobramysl wrote: > On Fri Feb 13 2015 at 11:07:39 PM Greg Ewing > wrote: > >> Ulrich Dobramysl wrote: >> >> > Thanks! I haven't figured out how to call heap allocated objects, as the >> > code >> > --- >> > cdef OperatorTest *t = new OperatorTest() >> > t() >> > --- >> > is not translatable by Cython. >> >> Have you tried: >> >> t[0]() >> >> ? >> > > Thanks, I didn't think about the t[0] trick! > > >> I haven't been following the development of Cython's internals, >> so I may be speaking naively here, but it looks wrong to me that >> the cname of that entry should be 'operator()' rather than the >> c-level name of the variable that the NameNode refers to. >> >> > The cname of the entry is correct IMO, because it cannot have any > knowledge about the cname of the object it is being called as later. The > logic dealing with the special case of an operator() member function of a > c++ class in SimpleCallNode.analyse_c_function_call() needs to be changed > slightly to not treat operator() as an overloaded entry. Rather, the > function type of the SimpleCallNode needs to be set to the type of the > looked-up operator() entry. Then, at the code generation stage, the correct > cname of the SimpleCallNode entry is used and not the entry of operator(). > With the patch below my test case compiles and runs without problems: > ---- > diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py > index f99ec6e..88522c9 100644 > --- a/Cython/Compiler/ExprNodes.py > +++ b/Cython/Compiler/ExprNodes.py > @@ -4569,11 +4569,13 @@ class SimpleCallNode(CallNode): > args = self.args > > if func_type.is_cpp_class: > - overloaded_entry = > self.function.type.scope.lookup("operator()") > - if overloaded_entry is None: > + call_operator = self.function.type.scope.lookup("operator()") > + if call_operator is None: > self.type = PyrexTypes.error_type > self.result_code = "" > return > + self.function.type = call_operator.type > + overloaded_entry = None > elif hasattr(self.function, 'entry'): > overloaded_entry = self.function.entry > elif (isinstance(self.function, IndexNode) and > @@ -4603,7 +4605,7 @@ class SimpleCallNode(CallNode): > else: > entry = None > func_type = self.function_type() > - if not func_type.is_cfunction: > + if not (func_type.is_cfunction or func_type.is_cpp_class): > error(self.pos, "Calling non-function type '%s'" % > func_type) > self.type = PyrexTypes.error_type > self.result_code = "" > ---- > What do you think? I don't know a lot about cython's internals, so there > might be use cases which break this code. > > Ulrich > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel What is the status on this? The most recent patch appears to work if the first two changed lines are ignored. If it would help move this along, I can open a PR with the modified patch and a commit that still attributes the patch to the original author. Thanks! -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From insertinterestingnamehere at gmail.com Tue Jun 23 04:25:22 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Tue, 23 Jun 2015 02:25:22 +0000 Subject: [Cython] operator() bug in cython In-Reply-To: References: Message-ID: On Mon, Jun 22, 2015 at 3:13 PM Ian Henriksen < insertinterestingnamehere at gmail.com> wrote: > On Tue, Feb 17, 2015 at 2:51 AM Ulrich Dobramysl wrote: > >> On Fri Feb 13 2015 at 11:07:39 PM Greg Ewing >> wrote: >> >>> Ulrich Dobramysl wrote: >>> >>> > Thanks! I haven't figured out how to call heap allocated objects, as >>> the >>> > code >>> > --- >>> > cdef OperatorTest *t = new OperatorTest() >>> > t() >>> > --- >>> > is not translatable by Cython. >>> >>> Have you tried: >>> >>> t[0]() >>> >>> ? >>> >> >> Thanks, I didn't think about the t[0] trick! >> >> >>> I haven't been following the development of Cython's internals, >>> so I may be speaking naively here, but it looks wrong to me that >>> the cname of that entry should be 'operator()' rather than the >>> c-level name of the variable that the NameNode refers to. >>> >>> >> The cname of the entry is correct IMO, because it cannot have any >> knowledge about the cname of the object it is being called as later. The >> logic dealing with the special case of an operator() member function of a >> c++ class in SimpleCallNode.analyse_c_function_call() needs to be changed >> slightly to not treat operator() as an overloaded entry. Rather, the >> function type of the SimpleCallNode needs to be set to the type of the >> looked-up operator() entry. Then, at the code generation stage, the correct >> cname of the SimpleCallNode entry is used and not the entry of operator(). >> With the patch below my test case compiles and runs without problems: >> ---- >> diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py >> index f99ec6e..88522c9 100644 >> --- a/Cython/Compiler/ExprNodes.py >> +++ b/Cython/Compiler/ExprNodes.py >> @@ -4569,11 +4569,13 @@ class SimpleCallNode(CallNode): >> args = self.args >> >> if func_type.is_cpp_class: >> - overloaded_entry = >> self.function.type.scope.lookup("operator()") >> - if overloaded_entry is None: >> + call_operator = self.function.type.scope.lookup("operator()") >> + if call_operator is None: >> self.type = PyrexTypes.error_type >> self.result_code = "" >> return >> + self.function.type = call_operator.type >> + overloaded_entry = None >> elif hasattr(self.function, 'entry'): >> overloaded_entry = self.function.entry >> elif (isinstance(self.function, IndexNode) and >> @@ -4603,7 +4605,7 @@ class SimpleCallNode(CallNode): >> else: >> entry = None >> func_type = self.function_type() >> - if not func_type.is_cfunction: >> + if not (func_type.is_cfunction or func_type.is_cpp_class): >> error(self.pos, "Calling non-function type '%s'" % >> func_type) >> self.type = PyrexTypes.error_type >> self.result_code = "" >> ---- >> What do you think? I don't know a lot about cython's internals, so there >> might be use cases which break this code. >> >> Ulrich >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel > > > What is the status on this? The most recent patch appears to work > if the first two changed lines are ignored. If it would help move this > along, I can open a PR with the modified patch and a commit that still > attributes the patch to the original author. > Thanks! > -Ian Henriksen > Alright, I've put up a pull request at https://github.com/cython/cython/pull/400. I hope this makes things a bit easier. -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sun Jun 28 21:06:50 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 28 Jun 2015 21:06:50 +0200 Subject: [Cython] generating documentation from Cython source code (was: [cython-users] Compiling a Cython module *without* having the required libraries) In-Reply-To: <559044C3.2060001@gmail.com> References: <55902388.7050508@behnel.de> <5590425D.6000704@behnel.de> <559044C3.2060001@gmail.com> Message-ID: <559045CA.6040706@behnel.de> Nikolaj van Omme schrieb am 28.06.2015 um 21:02: > On 15-06-28 02:52 PM, Stefan Behnel wrote: >> Nathann Cohen schrieb am 28.06.2015 um 18:44: >>>> Could you generate Python stub modules for them by grepping out def/cpdef >>>> functions and extension types together with their docstrings from the >>>> Cython sources? >>> Would there be a not-so-awful way to do this parsing? >>> >>> We may have classes and methods inside. Possibly several layers of it >>> (sorry, I know >_<). >> Well, the *right* way would let Cython do the parsing and integrate it with >> Sphinx, maybe via a Sphinx plugin. Either Cython could generate Python >> stubs (should be easy to do, see e.g. Cython/CodeWriter.py), or it could >> write out some XML/JSON/whatever representation of the source code metadata >> that other tools could use for further processing. >> >> The Python stub approach has the advantage of supporting normal Python >> tools, a dedicated file format has the advantage of allowing a more >> complete representation of the source code metadata. >> >> Both would be nice to have, I guess. But for the purpose of generating API >> documentation, Python stubs should get you most of the way already without >> requiring plugins for all documentation tools. Signature types could still >> be represented by mapping them to Python annotations (with a "# type: >> ignore" marker to disable PEP 484 type checks...). > > We are facing the same issues. Maybe we could team up and work on a > solution together? Please do. :) Can we move further discussion to the cython-devel mailing list? Stefan From stefan_ml at behnel.de Mon Jun 29 21:34:44 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 29 Jun 2015 21:34:44 +0200 Subject: [Cython] cython-0.23 monkey-patches stdlib? In-Reply-To: <55735732.9020403@behnel.de> References: <55735732.9020403@behnel.de> Message-ID: <55919DD4.5040500@behnel.de> Stefan Behnel schrieb am 06.06.2015 um 22:25: > Neal Becker schrieb am 06.06.2015 um 20:25: >> According to >> https://github.com/cython/cython/blob/master/CHANGES.rst >> >> the default build would attempt to modify python stdlib files? > > No. Glad you asked, that might not be clear. > > It will not modify the *files*, only the content of the modules after > importing them. It adds the missing ABCs to the collections.abc module and > registers Cython's own types with them *at runtime*. Here's what it does: > > https://github.com/cython/cython/blob/bb0dec2/Cython/Utility/Coroutine.c#L1322 > > This is a bit fragile, like all monkey-patching. As long as things get > imported in the right order, it works. But if you import asyncio before a > Cython module that adds the ABCs, it may already have decided that the ABCs > don't exist and won't use them. That's controllable on user application > side, though. > > Also, the first patcher necessarily wins, so if there will ever be updates > to the ABCs, old Cython modules would still patch in the old versions when > imported first. They should tend to be quite stable, though. > > In any case, this will get better over time as more code switches to Py3.5 > and later. Given the drawbacks above, I decided to extract the ABC patching into a separate PyPI module that users can install themselves to enable interoperability. https://pypi.python.org/pypi/backports_abc I'll remove the ABC patching code from Cython again. Stefan