From stefan_ml at behnel.de Sun Sep 1 12:17:25 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Sep 2013 12:17:25 +0200 Subject: [Cython] redesigning extension modules, avoiding static global C state Message-ID: <52231435.1060702@behnel.de> Hi, I recently started a discussion on python-dev to move this topic forward. http://thread.gmane.org/gmane.comp.python.devel/141262 Basically, the idea is to allow extension modules to implement their own module type, with all benefits like module level properties, proper garbage collection, sub-interpreter support, module reloading, etc. One outcome of that discussion (for me) was that a) there is an interest over there in doing that and b) it makes sense to start implementing this in Cython as far as we can already, meaning that we'd move all module globals into an extension type, turn global functions into methods of that type, move extension types to the heap and then hide that module type behind the existing module dict by copying over the references to its types and bound methods at module init time. PEP 3119 provides an infrastructure that we can use for now to get at least some of the benefits. Besides being some work to implement, I suspect that this also has a visible user impact, sadly. There might be a performance regression when we move from static globals to pointer indirections, often even multiple indirections. What would be more annoying is if we had to loosen some assumptions about globals that we currently make, e.g. regarding nogil safety or temping, as that could prevent users from doing stuff they currently do in their code. I don't remember if there were any major changes here for closures (which are similar), so not sure yet what the exact impact will be for globals. I think we should try to keep this out of the compiler itself as much as possible by using C macros for accessing globals. We'd still need to calculate the access path to the module state instance for each scope, but if we can move the rest to C compilation time, we may even be able to keep up a static-globals mode that modules could use if they value access time higher than more dynamic module support (e.g. because they cannot be reloaded or reinstantiated anyways due to external C dependencies not playing nicely). I'm not sure yet how much time I can invest into this during the next months, so if anyone else wants to join in for this, that'd be helpful. Stefan From robertwb at gmail.com Tue Sep 10 02:14:11 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 9 Sep 2013 17:14:11 -0700 Subject: [Cython] memoryviews bloat? Message-ID: >> Wow. The first version of this PR used Cython memoryviews, which added a whopping 14 kLOC of generated C to the repo. Switched back to bare pointers to keep the compile times within bounds. > It would be interesting to hear the Cython team's point of view on that. @sturlamolden @robertwb @markflorisson88 @dagss The priority has always been to produce the most optimized runtime code, with compile time being way down on the list of priorities, if even thought about at all. This means that there's likely a lot of low-hanging fruit for those who find the compile time too slow (do you have some specific stats?). Also, while Cython generates very verbose C code, much of it is optimized away. While the lines-of-code and compile times are related, they should not be conflated. The recently added common_utility_include_dir option to cythonize(...) might help as well--it allows utility code to be shared across multiple generated files. Perhaps a faster -c dbg option for development would be handy. - Robert From sturla at molden.no Tue Sep 10 10:39:17 2013 From: sturla at molden.no (Sturla Molden) Date: Tue, 10 Sep 2013 10:39:17 +0200 Subject: [Cython] memoryviews bloat? In-Reply-To: References: Message-ID: <8B54D2BE-0CD4-4AC6-A945-4A38FB280A68@molden.no> Den 10. sep. 2013 kl. 02:14 skrev Robert Bradshaw : >>> Wow. The first version of this PR used Cython memoryviews, which added a whopping 14 kLOC of generated C to the repo. Switched back to bare pointers to keep the compile times within bounds. >> It would be interesting to hear the Cython team's point of view on that. @sturlamolden @robertwb @markflorisson88 @dagss > > The priority has always been to produce the most optimized runtime > code, with compile time being way down on the list of priorities, if > even thought about at all. I thought perhaps the problem was the number of pyx files they had to compile. I.e. that the utility code is included in multiple C files, since they complained about 14 kLOC extra. But to that they answered: " larsmans commented a day ago The problem with typed memoryviews is simply that they're different from what we were doing. We'll have to change a lot of code and habits, being careful not to lose performance due to memoryview overhead." So I am not sure what the problem really was. Bloat or just an unfamiliar API? There was also a strange comment about tail-call optimization being prevented in cdef functions that are not declared nogil ? due to the refnanny. I am not sure what to make of that. Sturla -------------- next part -------------- An HTML attachment was scrubbed... URL: From python2 at dvadasz.hu Wed Sep 11 00:26:01 2013 From: python2 at dvadasz.hu (=?ISO-8859-1?Q?D=E9nes_Vad=E1sz?=) Date: Wed, 11 Sep 2013 00:26:01 +0200 Subject: [Cython] Metaclass bug corrupts the heap In-Reply-To: References: Message-ID: <522F9C79.6010309@dvadasz.hu> Hello, according to valgrind the following Cython fragment causes a heap corruption (with Python 2.7.5 and Cython 0.19.1 on Ubuntu): cdef class MetaClass(type): cdef int i class MyClass(object): __metaclass__ = MetaClass Please find below the result of a many hour investigation (originally triggered by random crashes and heap corruptions of a bigger Cython program that uses metaclasses). MetaClass is compiled to a C structure that extends PyTypeObject (a cc. 200 byte structure on a 32-bit architecture): struct __pyx_obj_4meta_MetaClass { PyTypeObject __pyx_base; int i; }; Instances of PyTypeObject are supposed to be allocated statically when initializing extension modules, because the structure does not support (among others) garbage collection. However, when MyClass is created, an instance of struct __pyx_obj_4meta_MetaClass (cc. 200 + 4 bytes) is dynamically allocated by the Python memory management machinery. The machinery then tries to initialize the allocated memory. The problem is that it expects that the first member of struct __pyx_obj_4meta_MetaClass is of type PyHeapTypeObject (a cc. 500 byte structure) and after a type cast it writes to the tail members of the assumed PyHeapTypeObject, i.e. way beyond the allocated cc. 200 + 4 bytes, corrupting the object heap. This corruption is nicely reported by valgrind. The proposed fix is to make __pyx_obj_4meta_MetaClass extend PyHeapTypeObject: struct __pyx_obj_4meta_MetaClass { PyHeapTypeObject __pyx_base; int i; }; This can be achieved by adding the below 2 lines to Compiler/Builtin.py: 382a383,384 > elif name == 'type': > objstruct_cname = 'PyHeapTypeObject' So that the init_builtin_types function becomes: def init_builtin_types(): global builtin_types for name, cname, methods in builtin_types_table: utility = builtin_utility_code.get(name) if name == 'frozenset': objstruct_cname = 'PySetObject' elif name == 'bool': objstruct_cname = None * elif name == 'type':* * objstruct_cname = 'PyHeapTypeObject'* else: objstruct_cname = 'Py%sObject' % name.capitalize() the_type = builtin_scope.declare_builtin_type(name, cname, utility, objstruct_cname) builtin_types[name] = the_type for method in methods: method.declare_in_type(the_type) After patching my Cython installation with the above, valgrind stopped to complain and there were no more crashes. Please consider adding this fix in the next release of Cython. Regards D?nes Vad?sz -------------- next part -------------- An HTML attachment was scrubbed... URL: From joonas.paalasmaa at gmail.com Thu Sep 12 07:46:43 2013 From: joonas.paalasmaa at gmail.com (Joonas Paalasmaa) Date: Thu, 12 Sep 2013 08:46:43 +0300 Subject: [Cython] Appending a dict to a struct vector creates invalid code Message-ID: Hello, I am appending Python dict objects to a C++ vector of structs. If it is done implicitly (calling v.push_back(dict_object)), Cython creates C++ code that does not compile. If dict_object has been cdef'd to be of the struct type, everything works. I am using version cf75e9eb7e from the master branch. Here is the code for reproducing the problem # cythonbug.pyx from libcpp.vector cimport vector ctypedef struct test_struct: int a int b def main(): cdef vector[test_struct] test_vector # cdef test_struct dict_object # Uncommenting this avoids the bug dict_object = {"a": 0, "b": 1} test_vector.push_back(dict_object) # end cythonbug.pyx The C++ output from "cython --cplus cythonbug.pyx" contains the statement "__pyx_t_9cythonbug_test_struct;" which GCC complains about: "error: declaration does not declare anything [-fpermissive]". Best wishes, Joonas Paalasmaa From stefan_ml at behnel.de Thu Sep 12 18:33:38 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Sep 2013 18:33:38 +0200 Subject: [Cython] memoryviews bloat? In-Reply-To: References: Message-ID: <5231ECE2.9000200@behnel.de> Robert Bradshaw, 10.09.2013 02:14: >>> Wow. The first version of this PR used Cython memoryviews, which added a whopping 14 kLOC of generated C to the repo. Switched back to bare pointers to keep the compile times within bounds. >> It would be interesting to hear the Cython team's point of view on that. @sturlamolden @robertwb @markflorisson88 @dagss Is there a link to this discussion? Stefan From sturla at molden.no Thu Sep 12 18:54:57 2013 From: sturla at molden.no (Sturla Molden) Date: Thu, 12 Sep 2013 18:54:57 +0200 Subject: [Cython] memoryviews bloat? In-Reply-To: <5231ECE2.9000200@behnel.de> References: <5231ECE2.9000200@behnel.de> Message-ID: On Sep 12, 2013, at 6:33 PM, Stefan Behnel wrote: > Robert Bradshaw, 10.09.2013 02:14: >>>> Wow. The first version of this PR used Cython memoryviews, which added a whopping 14 kLOC of generated C to the repo. Switched back to bare pointers to keep the compile times within bounds. >>> It would be interesting to hear the Cython team's point of view on that. @sturlamolden @robertwb @markflorisson88 @dagss > > Is there a link to this discussion? > https://github.com/scikit-learn/scikit-learn/pull/2426 Sturla From robertwb at gmail.com Thu Sep 12 23:02:03 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 12 Sep 2013 14:02:03 -0700 Subject: [Cython] Metaclass bug corrupts the heap In-Reply-To: <522F9C79.6010309@dvadasz.hu> References: <522F9C79.6010309@dvadasz.hu> Message-ID: Thanks for the careful analysis. See https://github.com/cython/cython/commit/e8b54dec4cdc94764abda4a107bc37dcd58e6f13 On Tue, Sep 10, 2013 at 3:26 PM, D?nes Vad?sz wrote: > Hello, > > according to valgrind the following Cython fragment causes a heap corruption > (with Python 2.7.5 and Cython 0.19.1 on Ubuntu): > > cdef class MetaClass(type): > cdef int i > > class MyClass(object): > __metaclass__ = MetaClass > > > > Please find below the result of a many hour investigation (originally > triggered by random crashes and heap corruptions of a bigger Cython program > that uses metaclasses). > > MetaClass is compiled to a C structure that extends PyTypeObject (a cc. 200 > byte structure on a 32-bit architecture): > > struct __pyx_obj_4meta_MetaClass { > PyTypeObject __pyx_base; > int i; > }; > > > Instances of PyTypeObject are supposed to be allocated statically when > initializing extension modules, because the structure does not support > (among others) garbage collection. However, when MyClass is created, an > instance of struct __pyx_obj_4meta_MetaClass (cc. 200 + 4 bytes) is > dynamically allocated by the Python memory management machinery. The > machinery then tries to initialize the allocated memory. The problem is that > it expects that the first member of struct __pyx_obj_4meta_MetaClass is of > type PyHeapTypeObject (a cc. 500 byte structure) and after a type cast it > writes to the tail members of the assumed PyHeapTypeObject, i.e. way beyond > the allocated cc. 200 + 4 bytes, corrupting the object heap. This corruption > is nicely reported by valgrind. > > The proposed fix is to make __pyx_obj_4meta_MetaClass extend > PyHeapTypeObject: > > struct __pyx_obj_4meta_MetaClass { > PyHeapTypeObject __pyx_base; > int i; > }; > > > This can be achieved by adding the below 2 lines to Compiler/Builtin.py: > > 382a383,384 >> elif name == 'type': >> objstruct_cname = 'PyHeapTypeObject' > > > So that the init_builtin_types function becomes: > > def init_builtin_types(): > global builtin_types > for name, cname, methods in builtin_types_table: > utility = builtin_utility_code.get(name) > if name == 'frozenset': > objstruct_cname = 'PySetObject' > elif name == 'bool': > objstruct_cname = None > elif name == 'type': > objstruct_cname = 'PyHeapTypeObject' > else: > objstruct_cname = 'Py%sObject' % name.capitalize() > the_type = builtin_scope.declare_builtin_type(name, cname, utility, > objstruct_cname) > builtin_types[name] = the_type > for method in methods: > method.declare_in_type(the_type) > > > After patching my Cython installation with the above, valgrind stopped to > complain and there were no more crashes. > > Please consider adding this fix in the next release of Cython. > > Regards > > D?nes Vad?sz > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Thu Sep 12 23:15:18 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Sep 2013 23:15:18 +0200 Subject: [Cython] Metaclass bug corrupts the heap In-Reply-To: References: <522F9C79.6010309@dvadasz.hu> Message-ID: <52322EE6.3090508@behnel.de> Robert Bradshaw, 12.09.2013 23:02: > Thanks for the careful analysis. See > https://github.com/cython/cython/commit/e8b54dec4cdc94764abda4a107bc37dcd58e6f13 Might be possible to merge it with this special case: https://github.com/cython/cython/blob/master/Cython/Compiler/Code.py#L50 and/or make it a part of BuiltinObjectType somehow. Stefan From linkmauve at linkmauve.fr Fri Sep 13 00:10:21 2013 From: linkmauve at linkmauve.fr (Emmanuel Gil Peyrot) Date: Fri, 13 Sep 2013 00:10:21 +0200 Subject: [Cython] Assign from array to scalar Message-ID: <20130912221021.GA9080@desktop> Hi, A few weeks ago I started an implementation of that simple usecase: cdef long a[2], x, y # set a[0] and a[1] to some useful values x, y = a It does works, but I?m not used to the interactions between the different parts of the code, I mainly hacked away where I could. I?m especially dissatisfied with the code duplication I came with (to check the correct types and size) and the location of the checks. I added testcases for both the array assignation and the slices of that same assignation I plan to implement, which seems to integrate badly with my current implementation, making the code even more unreadable. I tried to find a commit implementing a similar feature so I could get some inspiration but couldn?t find one. If someone has a hint about that I?d be glad. :) Btw, on a Cython.Compiler.ExprNodes.SliceIndexNode object, why does stop_code() give an int while start_code() gives a string? It?d be much better to return an int in both cases, so one could interpolate without conversion from both python or the generated C. Preliminary patch joined. -- Emmanuel Gil Peyrot XMPP: OpenPGP: 24B1D609 -------------- next part -------------- diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py index 2511eaf..bf04ff1 100755 --- a/Cython/Compiler/ExprNodes.py +++ b/Cython/Compiler/ExprNodes.py @@ -5615,6 +5615,11 @@ class SequenceNode(ExprNode): def generate_assignment_code(self, rhs, code): if self.starred_assignment: self.generate_starred_assignment_code(rhs, code) + elif (isinstance(self, TupleNode) and + isinstance(rhs, NameNode) and + isinstance(rhs.entry.type, PyrexTypes.CArrayType)): + self.generate_parallel_assignment_code2(rhs, code) + return else: self.generate_parallel_assignment_code(rhs, code) @@ -5627,6 +5632,22 @@ class SequenceNode(ExprNode): PyrexTypes.CFuncTypeArg("it", PyrexTypes.py_object_type, None), ])) + def generate_parallel_assignment_code2(self, rhs, code): + rhs_type = rhs.entry.type + base_type = rhs_type.base_type + size = rhs_type.size + lhs_args = self.args + assert len(lhs_args) == size + for arg in lhs_args: + if arg.entry.type is not base_type: + raise ValueError("Wrong type for parallel assignment of '%s' array: %s" % + (base_type, arg.entry.type)) + for i, arg in enumerate(lhs_args): + code.putln("%s = %s[%s];" % ( + arg.result(), + rhs.result(), + i)) + def generate_parallel_assignment_code(self, rhs, code): # Need to work around the fact that generate_evaluation_code # allocates the temps in a rather hacky way -- the assignment diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py index fe7adb3..7987cc6 100644 --- a/Cython/Compiler/Nodes.py +++ b/Cython/Compiler/Nodes.py @@ -4509,6 +4509,20 @@ class SingleAssignmentNode(AssignmentNode): self.lhs = self.lhs.analyse_target_types(env) self.lhs.gil_assignment_check(env) + if (isinstance(self.lhs, ExprNodes.TupleNode) and + isinstance(self.rhs, ExprNodes.NameNode)): + rhs_type = self.rhs.entry.type + if isinstance(rhs_type, PyrexTypes.CArrayType): + base_type = rhs_type.base_type + size = rhs_type.size + lhs_args = self.lhs.args + assert len(lhs_args) == size + for arg in lhs_args: + if arg.entry.type is not base_type: + break + else: + return self + if self.lhs.memslice_broadcast or self.rhs.memslice_broadcast: self.lhs.memslice_broadcast = True self.rhs.memslice_broadcast = True diff --git a/tests/run/arrayassign.pyx b/tests/run/arrayassign.pyx index 29c353e..f76ad2e 100644 --- a/tests/run/arrayassign.pyx +++ b/tests/run/arrayassign.pyx @@ -128,6 +128,59 @@ def test_ptr_literal_list_slice_end(): a[:5] = [1,2,3,4,5] return (a[0], a[1], a[2], a[3], a[4]) + +# The opposite, assignation from an array to multiple scalars. + +def test_array_to_scalars(): + """ + >>> test_array_to_scalars() + (1, 2, 3, 4, 5) + """ + cdef int l[5], a, b, c, d, e + l[:] = [1,2,3,4,5] + a, b, c, d, e = l + return (a, b, c, d, e) + +def test_slice_all_to_scalars(): + """ + >>> test_slice_all_to_scalars() + (1, 2, 3, 4, 5) + """ + cdef int l[5], a, b, c, d, e + l[:] = [1,2,3,4,5] + a, b, c, d, e = l[:] + return (a, b, c, d, e) + +def test_slice_start_to_scalars(): + """ + >>> test_slice_start_to_scalars() + (1, 2, 3, 4, 5) + """ + cdef int l[7], a, b, c, d, e + l[:] = [6,7,1,2,3,4,5] + a, b, c, d, e = l[2:] + return (a, b, c, d, e) + +def test_slice_end_to_scalars(): + """ + >>> test_slice_end_to_scalars() + (1, 2, 3, 4, 5) + """ + cdef int l[7], a, b, c, d, e + l[:] = [1,2,3,4,5,6,7] + a, b, c, d, e = l[:5] + return (a, b, c, d, e) + +def test_slice_start_end_to_scalars(): + """ + >>> test_slice_start_end_to_scalars() + (1, 2, 3, 4, 5) + """ + cdef int l[9], a, b, c, d, e + l[:] = [6,7,1,2,3,4,5,8,9] + a, b, c, d, e = l[2:7] + return (a, b, c, d, e) + # tuples aren't supported (yet) # #def test_literal_tuple(): -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From wesmckinn at gmail.com Mon Sep 16 20:48:56 2013 From: wesmckinn at gmail.com (Wes McKinney) Date: Mon, 16 Sep 2013 11:48:56 -0700 Subject: [Cython] Improving compile times for Cython extensions with C sources Message-ID: I haven't been able to find an easy answer to this question, but here is my problem: Cython file: extension.pyx Declarations / C imports: extension.pxd C dependencies: a.c b.c c.c These are all compiled together to produce extension.so. The problem is, when I modify extension.pyx, all 4 C files are recompiled, even if the 3 straight C files are untouched. Any way to fix the build system to not do this? Seems that the .o files from the C dependencies could be easily reused. thanks, Wes -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Mon Sep 16 22:12:00 2013 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 16 Sep 2013 13:12:00 -0700 Subject: [Cython] Improving compile times for Cython extensions with C sources In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 11:48 AM, Wes McKinney wrote: > I haven't been able to find an easy answer to this question, but here is my > problem: > > Cython file: extension.pyx > Declarations / C imports: extension.pxd > C dependencies: > a.c > b.c > c.c > > These are all compiled together to produce extension.so. > > The problem is, when I modify extension.pyx, all 4 C files are recompiled, > even if the 3 straight C files are untouched. > > Any way to fix the build system to not do this? Seems that the .o files from > the C dependencies could be easily reused. Cython compiles the .pyx file to a .c file; distutils takes over from there. I'm not aware of any way to do this (short of making a+b+c into a shared library) but perhaps someone on the distutils lists would have some ideas. You could also look at using ccache. - Robert From wesmckinn at gmail.com Tue Sep 17 21:09:06 2013 From: wesmckinn at gmail.com (Wes McKinney) Date: Tue, 17 Sep 2013 12:09:06 -0700 Subject: [Cython] Improving compile times for Cython extensions with C sources In-Reply-To: References: Message-ID: On Mon, Sep 16, 2013 at 1:12 PM, Robert Bradshaw wrote: > > On Mon, Sep 16, 2013 at 11:48 AM, Wes McKinney wrote: > > I haven't been able to find an easy answer to this question, but here is my > > problem: > > > > Cython file: extension.pyx > > Declarations / C imports: extension.pxd > > C dependencies: > > a.c > > b.c > > c.c > > > > These are all compiled together to produce extension.so. > > > > The problem is, when I modify extension.pyx, all 4 C files are recompiled, > > even if the 3 straight C files are untouched. > > > > Any way to fix the build system to not do this? Seems that the .o files from > > the C dependencies could be easily reused. > > Cython compiles the .pyx file to a .c file; distutils takes over from > there. I'm not aware of any way to do this (short of making a+b+c into > a shared library) but perhaps someone on the distutils lists would > have some ideas. > > You could also look at using ccache. > > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel Using ccache and patching distutils as described in this SO post was adequate for my needs: http://stackoverflow.com/questions/11013851/speeding-up-build-process-with-distutils thanks! From Bruno.Daniel at blue-yonder.com Wed Sep 18 15:14:06 2013 From: Bruno.Daniel at blue-yonder.com (Daniel, Bruno) Date: Wed, 18 Sep 2013 13:14:06 +0000 Subject: [Cython] Memory leak of memoryview attributes in cdef subclasses Message-ID: <2AECE7393BD56548A551D05DC076508818D0AA24@london.phi-tps.local> Dear developers, We encountered memory leaks when using memoryview attributes in cdef subclasses in Cython code. They can be avoided by adding a dealloc method setting the value of the memoryview attribute to None. The problem does not occur in topmost cdef classes. Here's an example: (See the module docstring on how to compile and run the example.) --- File memoryview_leak.pyx -------------------------------------------------- """ Cython module exposing memory leaks due to memoryview attributes in cdef classes Compile:: python setup.py build_ext --inplace Run:: python -c "import memoryview_leak; memoryview_leak.main()" """ from __future__ import print_function, division import unittest import gc import numpy as np cdef class LeakTestSuper: """ Superclass for the leak test in :class:`LeakTest1` and :class:`LeakTest2`. """ pass cdef class LeakTest1(LeakTestSuper): """ Class for memory leak testing. Holds a memoryview which is extracted from an :class:`numpy.ndarray` at initialization. This class avoids the leak by resetting the memoryview to None in the method :meth:`__dealloc__`. ``__del__()`` wouldn't work, because it is not supported in Cython's `cdef` classes. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) def __dealloc__(self): self._s = None cdef class LeakTest2(LeakTestSuper): """ Class for memory leak testing. Holds an array that is allocated at initialization. This class does not avoid the leak and thus exposes the problem with memoryviews in Cython. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) class TestLeakTest(unittest.TestCase): def n_objects_by_type(self, typename): return len([ obj for obj in gc.get_objects() if type(obj).__name__ == typename]) def test_gc(self): # Make sure to clear all old memoryviews (circularly referenced). gc.collect() # Now there mustn't be any memoryviews left. n_memory_views_expexted = 0 for cause_leak in [False, True, False, True]: # Create an object of LeakTest1 or LeakTest2 allocating a memoryview # internally. leaktest = LeakTest2() if cause_leak else LeakTest1() # Check the number of allocated memory views. n_memory_views_expexted += 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) # Delete the reference to leaktest and let the garbage collector do # its thing. del leaktest gc.collect() # Check for leaks by counting the memoryviews again. if not cause_leak: n_memory_views_expexted -= 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) def main(): unittest.main() ------------------------------------------------------------------------------- This is the corresponding setup file: --- File setup.py ------------------------------------------------------------- from __future__ import print_function, division from distutils.core import setup from Cython.Build import cythonize setup( name = "memoryview_leak", ext_modules = cythonize('memoryview_leak.pyx'), ) ------------------------------------------------------------------------------- If the unit tests are all ok in this example, this means that the memory leak has been encountered in LeakTest2 and avoided in LeakTest1 as expected. Best regards, Bruno Daniel ----- Dr. Bruno Daniel Research and Development Blue Yonder, Karlsruhe, Germany From pav at iki.fi Tue Sep 24 19:43:14 2013 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 24 Sep 2013 20:43:14 +0300 Subject: [Cython] Memory leak of memoryview attributes in cdef subclasses In-Reply-To: <2AECE7393BD56548A551D05DC076508818D0AA24@london.phi-tps.local> References: <2AECE7393BD56548A551D05DC076508818D0AA24@london.phi-tps.local> Message-ID: Hi, 18.09.2013 16:14, Daniel, Bruno kirjoitti: > We encountered memory leaks when using memoryview attributes in cdef subclasses > in Cython code. They can be avoided by adding a dealloc method setting the value > of the memoryview attribute to None. The problem does not occur in topmost > cdef classes. > > Here's an example: (See the module docstring on how to compile and run the > example.) Cannot reproduce the failure on Cython 0.19.1. If you are using an earlier version, please try that instead. One bug with memory leaks in cdef classes was fixed in 0.18 IIRC. -- Pauli Virtanen From Bruno.Daniel at blue-yonder.com Wed Sep 25 12:38:02 2013 From: Bruno.Daniel at blue-yonder.com (Daniel, Bruno) Date: Wed, 25 Sep 2013 10:38:02 +0000 Subject: [Cython] Memory leak of memoryview attributes in cdef subclasses Message-ID: <2AECE7393BD56548A551D05DC076508818D0E4AC@london.phi-tps.local> Dear Paul, > Cannot reproduce the failure on Cython 0.19.1. If you are using an > earlier version, please try that instead. We also used Cython 0.19.1. If the unit test runs through, this means that the memory leak is there. I'm sorry for the ambiguity. Please change the lines if not cause_leak: n_memory_views_expexted -= 1 into just n_memory_views_expexted -= 1 Then the test will fail if the memory leak exists. I also just noticed that unittest.main() doesn't work in Cython files. The tests are just ignored. Please call the tests this way instead: python -m unittest memoryview_leak For your convenience, here's the updated code: -------- File memoryview_leak.pyx --------------------------------------------------------------- """ Cython module exposing memory leaks due to memoryview attributes in cdef classes in Cython 0.19.1 Compile:: python setup.py build_ext --inplace Run:: python -m unittest memoryview_leak """ from __future__ import print_function, division import unittest import gc import numpy as np cdef class LeakTestSuper: """ Superclass for the leak test in :class:`LeakTest1` and :class:`LeakTest2`. """ pass cdef class LeakTest1(LeakTestSuper): """ Class for memory leak testing. Holds a memoryview which is extracted from an :class:`numpy.ndarray` at initialization. This class avoids the leak by resetting the memoryview to None in the method :meth:`__dealloc__`. ``__del__()`` wouldn't work, because it is not supported in Cython's `cdef` classes. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) def __dealloc__(self): self._s = None cdef class LeakTest2(LeakTestSuper): """ Class for memory leak testing. Holds an array that is allocated at initialization. This class does not avoid the leak and thus exposes the problem with memoryviews in Cython. """ cdef double[:] _s def __init__(self): self._s = np.empty(2) class TestLeakTest(unittest.TestCase): def n_objects_by_type(self, typename): return len([ obj for obj in gc.get_objects() if type(obj).__name__ == typename]) def test_gc(self): # Make sure to clear all old memoryviews (circularly referenced). gc.collect() # Now there mustn't be any memoryviews left. n_memory_views_expexted = 0 for cause_leak in [False, True, False, True]: # Create an object of LeakTest1 or LeakTest2 allocating a memoryview # internally. leaktest = LeakTest2() if cause_leak else LeakTest1() # Check the number of allocated memory views. n_memory_views_expexted += 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) # Delete the reference to leaktest and let the garbage collector do # its thing. del leaktest gc.collect() # Check for leaks by counting the memoryviews again. if True: # not cause_leak: n_memory_views_expexted -= 1 n_memory_views = self.n_objects_by_type('memoryview') self.assertEqual(n_memory_views, n_memory_views_expexted) ----------------------------------------------------------------------------------------------------- Here's the corresponding setup file: -------- File setup.py ---------------------------------------------------------------------------- from __future__ import print_function, division from distutils.core import setup from Cython.Build import cythonize setup( name = "memoryview_leak", ext_modules = cythonize('memoryview_leak.pyx'), ) --------------------------------------------------------------------------------------------------------- Best regards, Bruno Daniel From: cython-devel [cython-devel-bounces+bruno.daniel=blue-yonder.com at python.org] on behalf of cython-devel-request at python.org [cython-devel-request at python.org] Sent: Wednesday, September 25, 2013 12:00 PM To: cython-devel at python.org Subject: cython-devel Digest, Vol 32, Issue 10 Send cython-devel mailing list submissions to cython-devel at python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/cython-devel or, via email, send a message with subject or body 'help' to cython-devel-request at python.org You can reach the person managing the list at cython-devel-owner at python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of cython-devel digest..." Today's Topics: 1. Re: Memory leak of memoryview attributes in cdef subclasses (Pauli Virtanen) ---------------------------------------------------------------------- Message: 1 Date: Tue, 24 Sep 2013 20:43:14 +0300 From: Pauli Virtanen To: cython-devel at python.org Subject: Re: [Cython] Memory leak of memoryview attributes in cdef subclasses Message-ID: Content-Type: text/plain; charset=ISO-8859-1 Hi, 18.09.2013 16:14, Daniel, Bruno kirjoitti: > We encountered memory leaks when using memoryview attributes in cdef subclasses > in Cython code. They can be avoided by adding a dealloc method setting the value > of the memoryview attribute to None. The problem does not occur in topmost > cdef classes. > > Here's an example: (See the module docstring on how to compile and run the > example.) Cannot reproduce the failure on Cython 0.19.1. If you are using an earlier version, please try that instead. One bug with memory leaks in cdef classes was fixed in 0.18 IIRC. -- Pauli Virtanen ------------------------------ Subject: Digest Footer _______________________________________________ cython-devel mailing list cython-devel at python.org https://mail.python.org/mailman/listinfo/cython-devel ------------------------------ End of cython-devel Digest, Vol 32, Issue 10 ******************************************** From wstein at gmail.com Thu Sep 26 03:05:19 2013 From: wstein at gmail.com (William Stein) Date: Wed, 25 Sep 2013 18:05:19 -0700 Subject: [Cython] cython wiki Message-ID: Hello, I've permanently disabled the Cython wiki, due to all the spam it contains, and that whenever I try to start it, the moinmoin/twisted/python processes instantly just burns lots of CPU and stops responding to requests. If anybody would like to move the wiki to github, google code, bitbucket, etc., please send me an email and I will make the complete tarball of the wiki available to you. (wstein at gmail.com) -- William -- William Stein Professor of Mathematics University of Washington http://wstein.org From stefan_ml at behnel.de Sun Sep 29 09:39:29 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 29 Sep 2013 09:39:29 +0200 Subject: [Cython] Release plan for 0.20? Message-ID: <5247D931.6020109@behnel.de> Hi, I think 0.20 is shaping up nicely. Are there any further pending changes, or should we start preparing an alpha release? Stefan