From stefan_ml at behnel.de Mon Mar 3 17:38:58 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 03 Mar 2014 17:38:58 +0100 Subject: [Cython] Crash with freelist() and __slots__ Message-ID: <5314B022.3030407@behnel.de> Hi, I was made aware of crashes in the last lxml release, which turned out to be due to the use of freelists for types that could be subtyped from Python code. I was able to work around them in lxml, however, the real problem is in Cython. There was supposed to be a safe guard for that case in the freelist code based on the object struct size, which increases for subtypes that have a __dict__. However, if the Python subtype uses an empty __slots__ declaration, the object struct size will not increase, thus passing the guard. The correct fix is to also test if the type being instantiated lives on the heap and exclude it from the freelist if so. https://github.com/cython/cython/commit/d2aff824dd0900982a420ebaaa1dac6120a9d72e This, together with the buffer/memory view related bugs I fixed since the last release, suggest (at least to me) that we shouldn't wait all too long with the next bug fix release. >From my POV, all changes in current master are safe enough to go out. Stefan From robertwb at gmail.com Wed Mar 5 09:09:46 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 5 Mar 2014 00:09:46 -0800 Subject: [Cython] Crash with freelist() and __slots__ In-Reply-To: <5314B022.3030407@behnel.de> References: <5314B022.3030407@behnel.de> Message-ID: On Mon, Mar 3, 2014 at 8:38 AM, Stefan Behnel wrote: > Hi, > > I was made aware of crashes in the last lxml release, which turned out to > be due to the use of freelists for types that could be subtyped from Python > code. I was able to work around them in lxml, however, the real problem is > in Cython. There was supposed to be a safe guard for that case in the > freelist code based on the object struct size, which increases for subtypes > that have a __dict__. However, if the Python subtype uses an empty > __slots__ declaration, the object struct size will not increase, thus > passing the guard. > > The correct fix is to also test if the type being instantiated lives on the > heap and exclude it from the freelist if so. > > https://github.com/cython/cython/commit/d2aff824dd0900982a420ebaaa1dac6120a9d72e > > This, together with the buffer/memory view related bugs I fixed since the > last release, suggest (at least to me) that we shouldn't wait all too long > with the next bug fix release. Sounds good to me. Did you want to get the process rolling, or should I? > From my POV, all changes in current master are safe enough to go out. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Wed Mar 5 18:06:53 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 05 Mar 2014 18:06:53 +0100 Subject: [Cython] Crash with freelist() and __slots__ In-Reply-To: References: <5314B022.3030407@behnel.de> Message-ID: <531759AD.1030303@behnel.de> Robert Bradshaw, 05.03.2014 09:09: > On Mon, Mar 3, 2014 at 8:38 AM, Stefan Behnel wrote: >> we shouldn't wait all too long with the next bug fix release. > Sounds good to me. Did you want to get the process rolling, or should I? I wouldn't dare to jump the queue. :) Stefan From pav at iki.fi Sat Mar 8 21:11:55 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 08 Mar 2014 22:11:55 +0200 Subject: [Cython] Too many instantiations with fused type memoryviews Message-ID: Hi, FYI: Cython instantiates fused type routines with memoryview arguments unnecessarily many times. Example: ``` ctypedef fused idx_t: short long # Cython 0.20.1 instantiates this function 128 times, # even though only 2 would be needed def fubar_128(idx_t[:] a, idx_t[:] b, idx_t[:] c, idx_t[:] d, idx_t[:] e, idx_t[:] f, idx_t[:] g): print("HALLO") def fubar_2(idx_t a, idx_t b, idx_t c, idx_t d, idx_t e, idx_t f, idx_t g): print("HULLO") ``` $ cython cymod.pyx $ cython --version Cython version 0.20.1 $ grep -c 'print("HALLO")' cymod.c 128 $ grep -c 'print("HULLO")' _cymod.c 2 The n**m explosion starts to hurt quite quickly when there are several array arguments and more than one fused type. I think this issue is also accompanied by some signature resolution bugs (I'll try to come up with an example case). Cheers, -- Pauli Virtanen From stefan_ml at behnel.de Sat Mar 8 21:40:40 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 08 Mar 2014 21:40:40 +0100 Subject: [Cython] Too many instantiations with fused type memoryviews In-Reply-To: References: Message-ID: <531B8048.3060404@behnel.de> Pauli Virtanen, 08.03.2014 21:11: > FYI: Cython instantiates fused type routines with memoryview arguments > unnecessarily many times. > > Example: > ``` > ctypedef fused idx_t: > short > long > > # Cython 0.20.1 instantiates this function 128 times, > # even though only 2 would be needed > def fubar_128(idx_t[:] a, > idx_t[:] b, > idx_t[:] c, > idx_t[:] d, > idx_t[:] e, > idx_t[:] f, > idx_t[:] g): > print("HALLO") > > def fubar_2(idx_t a, > idx_t b, > idx_t c, > idx_t d, > idx_t e, > idx_t f, > idx_t g): > print("HULLO") > ``` > $ cython cymod.pyx > $ cython --version > Cython version 0.20.1 > $ grep -c 'print("HALLO")' cymod.c > 128 > $ grep -c 'print("HULLO")' _cymod.c > 2 > > The n**m explosion starts to hurt quite quickly when there are several > array arguments and more than one fused type. I think this issue is > also accompanied by some signature resolution bugs (I'll try to come > up with an example case). Yes, this is a known problem. The work-around is to ctypedef the memory view type, not just the dtype. http://thread.gmane.org/gmane.comp.python.cython.user/10437 Stefan From pav at iki.fi Sat Mar 8 21:59:18 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 08 Mar 2014 22:59:18 +0200 Subject: [Cython] Too many instantiations with fused type memoryviews In-Reply-To: <531B8048.3060404@behnel.de> References: <531B8048.3060404@behnel.de> Message-ID: 08.03.2014 22:40, Stefan Behnel kirjoitti: [clip] > Yes, this is a known problem. The work-around is to ctypedef the > memory view type, not just the dtype. > > http://thread.gmane.org/gmane.comp.python.cython.user/10437 Thanks, I missed that. Re: arguments in the previous thread --- I think it would indeed be better if memoryviews would not invoke cross-product behavior. If scalars work in one way, also arrays should work in the same way. Pauli From scopatz at gmail.com Wed Mar 12 19:55:42 2014 From: scopatz at gmail.com (Anthony Scopatz) Date: Wed, 12 Mar 2014 13:55:42 -0500 Subject: [Cython] Numpy Deprecation Warnings In-Reply-To: References: Message-ID: Forwarding this on to cython developers. On Sun, Mar 9, 2014 at 9:54 AM, Anthony Scopatz wrote: > Hello All, > > I continue to see numpy deprecation warnings. When I set the right flag > flag, the warnings go away but then the build fails because it seems Cython > *is* generating to the old API even though I am not using it! Is this what > is actually going on? And if so, is there a fix in the pipeline? > > Be Well > Anthony > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Mar 13 07:39:41 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 13 Mar 2014 07:39:41 +0100 Subject: [Cython] Numpy Deprecation Warnings In-Reply-To: References: Message-ID: <532152AD.5070700@behnel.de> [copying my answer from cython-users, which the core developers tend to read as well] Anthony Scopatz, 12.03.2014 19:55: > Forwarding this on to cython developers. > On Sun, Mar 9, 2014 at 9:54 AM, Anthony Scopatz wrote: >> I continue to see numpy deprecation warnings. When I set the right flag >> flag, the warnings go away but then the build fails because it seems Cython >> *is* generating to the old API even though I am not using it! Is this what >> is actually going on? And if so, is there a fix in the pipeline? No fix prepared, but we certainly take patches. Stefan From pav at iki.fi Tue Mar 18 23:56:36 2014 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 19 Mar 2014 00:56:36 +0200 Subject: [Cython] Fused type signature resolution failures Message-ID: Hi, Here are the fused types + memoryview signature resolution failures I promised earlier (Cython 0.20.1): (A) TypeError: No matching signature ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused value_t: cnp.float32_t cnp.float64_t cpdef foo(value_t x): pass ----------------------------- quux.py import numpy as np import asd asd.foo(np.float32(1.0)) ----------------------------- (B) ValueError: Buffer dtype mismatch, expected 'int64_t' but got 'double' ----------------------------- asd.pyx cimport numpy as cnp ctypedef fused idx_t: cnp.int32_t cnp.int64_t ctypedef fused value_t: cnp.int64_t cnp.float64_t cpdef foo(idx_t[:,:] i, idx_t[:,:] j, value_t[:,:] x): pass ----------------------------- quux.py import numpy as np import asd i = np.zeros((3, 3), np.int64) j = np.zeros((3, 3), np.int64) x = np.zeros((3, 3), np.float64) asd.foo(i, j, x) ----------------------------- (C) Then some nasty platform-dependent failures: https://github.com/scipy/scipy/issues/3461 The relevant code is: https://github.com/scipy/scipy/blob/master/scipy/sparse/_csparsetools.pyx#L202 The code looks nothing special. However, call to `lil_fancy_get` fails with "TypeError: No matching signature found" when the inputs have types with ndarray dtypes: object object object object int32 int32 The failure occurs only on Christoph Gohlke's Win64 build, but not on Linux/OSX/MINGW32. This sounds like some integer size combination issue, but I'm far from sure. Unfortunately, I'm not easily able to isolate/understand what's going wrong here. -- Pauli Virtane From stefan_ml at behnel.de Fri Mar 21 08:07:46 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 21 Mar 2014 08:07:46 +0100 Subject: [Cython] 0.20.2 In-Reply-To: <530867A2.4090604@behnel.de> References: <530867A2.4090604@behnel.de> Message-ID: <532BE542.60500@behnel.de> Stefan Behnel, 22.02.2014 10:02: > there are a couple of nice additions and fixes in the current master that > could go out as a 0.20.2 some time soon. I suggest waiting another couple > of days for more bug reports to come in and then just send it out. I merged the current master over into 0.20.x. The majority of changes were bug fixes, and the rest wasn't worth putting back for the next major release IMHO. I suggest we apply further fixes directly to the release branch from now on and move on to 0.21 with the master branch. Stefan From robertwb at gmail.com Fri Mar 21 17:03:33 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 21 Mar 2014 09:03:33 -0700 Subject: [Cython] 0.20.2 In-Reply-To: <532BE542.60500@behnel.de> References: <530867A2.4090604@behnel.de> <532BE542.60500@behnel.de> Message-ID: On Fri, Mar 21, 2014 at 12:07 AM, Stefan Behnel wrote: > Stefan Behnel, 22.02.2014 10:02: >> there are a couple of nice additions and fixes in the current master that >> could go out as a 0.20.2 some time soon. I suggest waiting another couple >> of days for more bug reports to come in and then just send it out. > > I merged the current master over into 0.20.x. The majority of changes were > bug fixes, and the rest wasn't worth putting back for the next major > release IMHO. > > I suggest we apply further fixes directly to the release branch from now on > and move on to 0.21 with the master branch. Sounds good. There are a couple of bugs that have come up (e.g. the struct conversion one) that I'd like to get fixed before pushing out this bug fix release, though I haven't had time to look into them deeply enough yet. I'll push any such work on the 0.20.x release branch. - Robert From sturla.molden at gmail.com Sat Mar 22 03:33:05 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Sat, 22 Mar 2014 02:33:05 +0000 (UTC) Subject: [Cython] Too many instantiations with fused type memoryviews References: Message-ID: <1656996925417148047.911386sturla.molden-gmail.com@news.gmane.org> Pauli Virtanen wrote: > The n**m explosion starts to hurt quite quickly when there are several > array arguments and more than one fused type. I think this issue is > also accompanied by some signature resolution bugs (I'll try to come > up with an example case). I warned that fused types would be a bloatware generator. This behavior is "by design". https://mail.python.org/pipermail/cython-devel/2011-May/000670.html https://mail.python.org/pipermail/cython-devel/2011-May/000648.html Sturla From stefan_ml at behnel.de Sat Mar 22 11:08:56 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 22 Mar 2014 11:08:56 +0100 Subject: [Cython] Too many instantiations with fused type memoryviews In-Reply-To: References: <531B8048.3060404@behnel.de> Message-ID: <532D6138.4040009@behnel.de> Pauli Virtanen, 08.03.2014 21:59: > 08.03.2014 22:40, Stefan Behnel kirjoitti: > [clip] >> Yes, this is a known problem. The work-around is to ctypedef the >> memory view type, not just the dtype. >> >> http://thread.gmane.org/gmane.comp.python.cython.user/10437 > > Thanks, I missed that. > > Re: arguments in the previous thread --- I think it would indeed be > better if memoryviews would not invoke cross-product behavior. If > scalars work in one way, also arrays should work in the same way. I'm not a fused types user myself, but it seems to me that it's worth changing this behaviour for 0.21 to better match the most common use cases. A pull request that implements this certainly won't hurt, if only to get real-world feedback on the change. The work-around for (existing) code that actually wants the cross product behaviour for memory views (for whatever reason) would then be (I guess) to use two identical typedefs with different names. This sounds acceptable, given that the cross product suggests that we are dealing with independent (i.e. different) kinds of data anyway. Stefan From pav at iki.fi Sat Mar 22 13:48:13 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 22 Mar 2014 14:48:13 +0200 Subject: [Cython] Too many instantiations with fused type memoryviews In-Reply-To: <532D6138.4040009@behnel.de> References: <531B8048.3060404@behnel.de> <532D6138.4040009@behnel.de> Message-ID: Hi, 22.03.2014 12:08, Stefan Behnel kirjoitti: [clip] >> Re: arguments in the previous thread --- I think it would indeed be >> better if memoryviews would not invoke cross-product behavior. If >> scalars work in one way, also arrays should work in the same way. > > I'm not a fused types user myself, but it seems to me that it's worth > changing this behaviour for 0.21 to better match the most common use cases. > A pull request that implements this certainly won't hurt, if only to get > real-world feedback on the change. > > The work-around for (existing) code that actually wants the cross product > behaviour for memory views (for whatever reason) would then be (I guess) to > use two identical typedefs with different names. This sounds acceptable, > given that the cross product suggests that we are dealing with independent > (i.e. different) kinds of data anyway. I think another argument against the cross-product behavior is that constructs such as cdef idx_t foo and x = y and if idx_t is np.int: # FOO else: # BAR all become ambiguous in the function body, if idx_t is instantiated in several different ways. I can take a look at this today... (Re: Sturla: I *want* a bloatware generator, but a controllable one :) -- Pauli Virtanen From pav at iki.fi Sat Mar 22 19:37:52 2014 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 22 Mar 2014 20:37:52 +0200 Subject: [Cython] Too many instantiations with fused type memoryviews In-Reply-To: References: <531B8048.3060404@behnel.de> <532D6138.4040009@behnel.de> Message-ID: 22.03.2014 14:48, Pauli Virtanen kirjoitti: [clip] > I can take a look at this today... Here's one attempt: https://github.com/cython/cython/pull/284 In the long run, I'd like to be able to write code like this: https://github.com/pv/scipy-work/blob/csparsetools-nocrud/scipy/sparse/_csparsetools.pyx with fused types and have it just work. The above change gets it mostly towards the goal, apart from issues with Numpy scalars and numpy boolean arrays. I'll probably look at this later on... -- Pauli Virtanen From syam.gadde at duke.edu Thu Mar 27 19:10:17 2014 From: syam.gadde at duke.edu (Syam Gadde) Date: Thu, 27 Mar 2014 18:10:17 +0000 Subject: [Cython] line tracing/profiling code objects In-Reply-To: <530FA8DE.9030200@behnel.de> References: <530F5822.1070202@duke.edu> <530FA8DE.9030200@behnel.de> Message-ID: <53346989.8030302@duke.edu> Stefan, I'm sorry to be so long in responding. Your response was very helpful, but I had to suddenly change my focus to other projects. I've been looking at it again. I took the easy way out for now and attempted to name the codeobj objects deterministically, as you suggested at the end of your message: https://github.com/SyamGadde/cython/commit/b56135154f546eec34d88342dc1698cf1803492e The assumption is that only one codeobj is ever generated for each named function within a module. I'm afraid I may have taken a hammer to the code instead of a chisel, so your comments are welcome. This is actually sufficient for using the kernprof/line_profiler decorator @profile as-is on Cython-generated non-cdef functions/methods as long as you are using the following cython directives: --directive profile=true --directive linetrace=true --directive binding=true The binding=true is necessary so that the function objects get the func_code attribute set to the codeobj above. Actually line_profiler just uses func_code as a unique identifier for the function and doesn't really look at the contents of func_code except to determine if a function is a generator. If there is another attribute available for uniquely identifying function objects that don't have func_code, then line_profiler could potentially use that too and it could work on cdef- and cpdef-declared functions... ...except for the fact that cdef and cpdef functions don't accept arbitrary decorators, which is the typical way to use kernprof/line_profiler. But I'm pretty happy to be able to use kernprof/line_profiler on compiled pure-Python functions for now. Thanks for your help! -syam On 02/27/2014 04:06 PM, Stefan Behnel wrote: > Hi! > > Syam Gadde, 27.02.2014 16:22: >> I tried using line tracing in conjunction with line_profiler/kernprof to >> attempt to see if line-based profiling works in Cython. I think it's >> pretty close. But I think the problem is that line_profiler is getting >> a different PyCodeObject when it wraps the function and when it actually >> gets the trace call. It adds the initial code object to a map, and >> later when it gets the trace call, decides that the trace call is not >> something it needs to pay attention to because the new code object that >> it gets is not the same as the original one. >> >> The first code object is created at function declaration by >> __Pyx_PyCode_New (called in__Pyx_InitCachedConstants) and is assigned to >> a variable __pyx_k_codeobj_NNN. The second code object is created, >> essentially, during the first entry into the function (in >> __Pyx_TraceCall, via __Pyx_TraceSetupAndCall). It seems that setting >> __pyx_frame_code to the initial code object before calling TraceCall() >> would fix this. > That's a part of it, yes. Here's another bit in the puzzle: > > https://github.com/cython/cython/pull/93 > > The problem is that the code objects that we currently create along the way > have a fixed Python code line number (because that was the simplest way to > get it working). The two changes above will get us pretty far. What remains > to be done then is to enable line number calculation at runtime by > emulating byte code offsets in the frame object instead of using absolute > line numbers. The pull request refers to CPython's > Objects/lnotab_notes.txt, which has some details on the inner workings. > > Properly calculating line numbers at runtime would also have other nice > side effects, because it would remove the need to create multiple code > objects for a single function in the first place. So it's very desirable. > > >> Is this easy to do? I'd do it myself, but I'd need help figuring out >> how to get the name of the appropriate __pyx_k_codeobj_NNN variable from >> within FuncDefNode.generate_function_definitions(), which calls >> put_trace_call(). > The best way is usually to run it through a debugger and see what you can > get your hands on. :) The constant names (__pyx_k_...) are generated on the > fly at C code generation time, so you can't get them before that. Two > possible solutions: either change the way how the C names of code objects > are being generated (e.g. by making their C name depend on the mangled C > name of the function so that they can be deterministically named at > analysis time), or make the code object node remember its generated > constant name and reuse it in other places. > > As you can see, it's not entirely trivial overall, but we've already been > piling up the bits and pieces so that the only remaining effort is to put > everything together. If you could give it a try, I'm sure you'd make a lot > of people happy with it. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From syam.gadde at duke.edu Thu Mar 27 21:29:20 2014 From: syam.gadde at duke.edu (Syam Gadde) Date: Thu, 27 Mar 2014 20:29:20 +0000 Subject: [Cython] memoryview slice transpose Message-ID: <53348A20.7020305@duke.edu> I wanted to re-offer for discussion a fix for a bug I reported a while back. The code that breaks is this: import numpy cimport numpy def testfunc(): a = numpy.arange(12).reshape([3,4]) cdef numpy.int_t[:,:] a_view = a cdef numpy.int_t[:,:] b = a_view[:,:].T One problem is that when a memoryview slice is decremented, the .memview and .data fields are not set to NULL, and any error that causes a jump to cleanup code will attempt to decref it again. The second problem (I think) is that assignment of a transpose of memoryview slices to a variable is not resulting in an incref on the underlying slice, even though the variable is not a temporary and continues to persist. Here is a patch that solves the problem for me. https://github.com/SyamGadde/cython/commit/5739d8b908f18c4fc9103ef04e39964d61af3495 The part I am least sure about is removing the "if" qualifications on the incref and decref: if self.obj.is_name or self.obj.is_attribute and self.obj.is_memslice_transpose: that was obviously there for a reason, but it causes problems in the above case. If there is a less extreme solution I'd be happy to take it. Thanks, -syam