From stefan_ml at behnel.de Wed May 7 17:40:31 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 07 May 2014 17:40:31 +0200 Subject: [Cython] Bug in C code generation In-Reply-To: References: Message-ID: <536A53EF.5060905@behnel.de> Ulf Worsoe, 30.04.2014 12:45: > I have encounterered a bug in the following situation with Cython 0.20.1: > > cdef funname(): > cdef numpy.ndarray[numpy.int32_t,ndim=1] myvar > if True: > ... block that never initializes myvar > else > ... block that initializes myvar > > The resulting C code does not declare the variable for the buffer object of > myvar (my guess is that cython detects that it is never used and drops it), > but the cleanup part of the C function contains code that frees the buffer, > and thus refers to a variable that was never declared. I tried the following and it works for me, the buffer code gets properly discarded: ''' cimport numpy as np def disabled_usage(obj): cdef object[int, ndim=2] buf cdef np.ndarray[np.int32_t, ndim=1] buf2 if False: buf = obj elif not True: buf2 = obj return obj ''' Could you try to find a minimal example that shows the problem? Stefan From redbrain at gcc.gnu.org Fri May 9 17:09:50 2014 From: redbrain at gcc.gnu.org (Philip Herron) Date: Fri, 9 May 2014 16:09:50 +0100 Subject: [Cython] GCC Pxd In-Reply-To: <536245A4.4090602@behnel.de> References: <536245A4.4090602@behnel.de> Message-ID: Just small update i think i have got most of C code working now, but working on C++ generation again its all shaping up to look pretty usable. I wonder by the off chance is there a python library that can parse the C preprocessor if anyone knows or not i am looking at https://code.google.com/p/pypreprocessor/ to try and parse cpp as this information is lost at the Middle-end level in GCC where the plugin runs. On 1 May 2014 14:01, Stefan Behnel wrote: > [forwarding Philip's mail to the Cython users mailing list] > > Hey all, > > Some of you may remember a project i worked on for GSoC 2012, i wasn't > really happy with the project over all for a long time it basicaly > required at the time people to be running GCC from latest SVN/GIT for > it to work correctly. > > Now gcc >= 4.7 is becoming more normal, so i went back into it and > rewrote it from scratch now and its shaping up to look fairly decent > and usable now. > > https://github.com/redbrain/cython/tree/master/Tools/gccpxd > > So the idea for this project is that you write a small configuration > file and it will integrate with your C/C++ build system for example > you could do: > > ./configure CC=cygcc ... > make > ... > > And you will get a project.pxd file after the build of your software. > the cygcc is simply a wrapper over GCC that shoves in the fplugin > arguments so you can run: > > cygcc -g -O2 -Wall -c test.c -o test.o > > So build systems wont really notice anything different it should just > think its gcc. The configuration file example in the folder .cygcc: > > $ cat .cygcc > header = 'test.h' # the header to get declaratiosn for > inputs = ['test.h', 'test.c'] # any locations of declarations that > would be involved > output = 'test.pxd' # the output > > The cygcc compiler looks for this in the current director or looks up > at the parent directory untill it finds it or it will fail with an > error message. > > It will only ouput once if the test.pxd file exists in the current > working directory it wont do anything. > > $ cat test.pxd > cdef extern from "test.h": > int test (int, int) > int global_variable > int static_variable # this is a bug but hey... > struct bla: > int a > float b > > So far for the example code in there its working pretty well the gcc > python plugin has stabalized alot and so has some of gcc's bits a > bobs. I will try and keep you all updated i think the plugin is doing > alot more work now so this doesnt have to do so much. > > Thanks to Robert Bradshaw for baring with me in the whole thing > because i probably wasn't the easiest to work with ;) lol. > > Thanks everyone cython is awesome. > > --Phil > From joshua.adelman at gmail.com Tue May 20 22:25:01 2014 From: joshua.adelman at gmail.com (Joshua Adelman) Date: Tue, 20 May 2014 16:25:01 -0400 Subject: [Cython] undefined reference to `__Pyx_PyInt_As_unsigned_int error Message-ID: <27537EB6-6DA7-499D-8A29-502663C20325@gmail.com> Hello all, I wanted to try to get some traction on this issue that I raised on the Cython-user google group list: https://groups.google.com/forum/#!topic/cython-users/l9wD4w2lcw8 The basic issue is that I get an error of "undefined reference to `__Pyx_PyInt_As_unsigned_int". The strange part is I then get non-deterministic codegen, wherein the first time I compile the code it fails, but if I delete the .c file and try again, it will then compile without errors. A minimal example is provided here: https://gist.github.com/synapticarbors/11060931 This issue seems to have cropped up somewhere between 0.19.2 (where it's fine) and 0.20.1. It also is still present in 0.21dev from github. The error seems to happen when running on a Windows machine (Windows 2008 server), but I can't reproduce the error on Mac OSX (running the same version of the Continuum Anaconda distribution). Any suggestions or confirmation of this issue would be appreciated. Thanks, Josh -------------- next part -------------- An HTML attachment was scrubbed... URL: From linkmauve at linkmauve.fr Wed May 28 13:25:47 2014 From: linkmauve at linkmauve.fr (Emmanuel Gil Peyrot) Date: Wed, 28 May 2014 13:25:47 +0200 Subject: [Cython] =?utf-8?q?Wrong_order_of_=5F=5FPyx=5FDECREF_when_calling?= =?utf-8?q?_a_function_with_an_implicit_str_=E2=86=92_char*_conversion=2E?= Message-ID: <20140528112547.GA2814@yuyuko> Hi, I was testing my cython codebase on top of pypy/cpyext, and I found a memory corruption. After investigating it with the pypy guys (thanks arigato!), we identified it as a cython bug: cdef void test(char *string): print(string) def run(array): test(array[0]) This code works fine under cpython, but looking at the generated code for the last line, the call to test is done with a pointer (__pyx_t_2) to some potentially deallocated string (__pyx_t_1), which isn?t freed just yet on cpython but is on pypy: __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, ?); __Pyx_GOTREF(__pyx_t_1); __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1); __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0; __pyx_f_1a_test(__pyx_t_2); The obvious solution is to swap the last two lines, it then works fine on pypy (although not necessarily if the test function stores the pointer somewhere, but it?s not cython?s fault then). This issue can also happen with an explicit cast: pointer = array[0] test(pointer) I?m not sure if it should be addressed the same way, because that would mean keeping a reference to array[0] for all the lifetime of the current scope, but it could still prevent obscure bugs like the memory corruption I had. -- Emmanuel Gil Peyrot -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From robertwb at gmail.com Fri May 30 03:06:04 2014 From: robertwb at gmail.com (Robert Bradshaw) Date: Thu, 29 May 2014 18:06:04 -0700 Subject: [Cython] Cython 0.20.2 beta Message-ID: I've pushed a beta for the 0.20.2 can be found at http://cython.org/release/Cython-0.20rc1.tar.gz . This is a bugfix-only release, more development is going on for 0.21 which will also be released soon. Note, however, that the 0.20.x branch is the last to support older versions of Python (pre 2.6 or 3.2). Please try it out and report back. - Robert Features added Some optimisations for set/frozenset instantiation. Support for C++ unordered_set and unordered_map. Bugs fixed Access to attributes of optimised builtin methods (e.g. [].append.__name__) could fail to compile. Memory leak when extension subtypes add a memory view as attribute to those of the parent type without having Python object attributes or a user provided dealloc method. Compiler crash on readonly properties in "binding" mode. Auto-encoding with c_string_encoding=ascii failed in Py3.3. Crash when subtyping freelist enabled Cython extension types with Python classes that use __slots__. Freelist usage is restricted to CPython to avoid problems with other Python implementations. Memory leak in memory views when copying overlapping, contiguous slices. Format checking when requesting non-contiguous buffers from cython.array objects was disabled in Py3. C++ destructor calls in extension types could fail to compile in clang. Buffer format validation failed for sequences of strings in structs. Docstrings on extension type attributes in .pxd files were rejected. From stefan_ml at behnel.de Fri May 30 15:01:18 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 30 May 2014 15:01:18 +0200 Subject: [Cython] =?utf-8?q?Wrong_order_of_=5F=5FPyx=5FDECREF_when_calling?= =?utf-8?q?_a_function_with_an_implicit_str_=E2=86=92_char*_conversion=2E?= In-Reply-To: <20140528112547.GA2814@yuyuko> References: <20140528112547.GA2814@yuyuko> Message-ID: <5388811E.3090802@behnel.de> Emmanuel Gil Peyrot, 28.05.2014 13:25: > I was testing my cython codebase on top of pypy/cpyext, and I found a > memory corruption. After investigating it with the pypy guys (thanks > arigato!), we identified it as a cython bug: > > cdef void test(char *string): > print(string) > > def run(array): > test(array[0]) > > > This code works fine under cpython, but looking at the generated code > for the last line, the call to test is done with a pointer (__pyx_t_2) > to some potentially deallocated string (__pyx_t_1), which isn?t freed > just yet on cpython but is on pypy: > > __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, ?); > __Pyx_GOTREF(__pyx_t_1); > __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1); > __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0; > __pyx_f_1a_test(__pyx_t_2); > > > The obvious solution is to swap the last two lines, it then works fine > on pypy (although not necessarily if the test function stores the > pointer somewhere, but it?s not cython?s fault then). > > This issue can also happen with an explicit cast: > > pointer = array[0] > test(pointer) > > > I?m not sure if it should be addressed the same way, because that would > mean keeping a reference to array[0] for all the lifetime of the > current scope, but it could still prevent obscure bugs like the memory > corruption I had. Neither of these two examples should compile. Even in CPython, indexing can not only return a safe reference but a new object, e.g. class Indy(object): def __getitem__(self, i): return "abc%s" % i run(Indy()) This should crash also in CPython, or at least show unpredictable results. Cython has a mechanism to reject this kind of code, not sure why it wouldn't strike here. Stefan From stefan_ml at behnel.de Fri May 30 15:24:55 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 30 May 2014 15:24:55 +0200 Subject: [Cython] =?utf-8?q?Wrong_order_of_=5F=5FPyx=5FDECREF_when_calling?= =?utf-8?q?_a_function_with_an_implicit_str_=E2=86=92_char*_conversion=2E?= In-Reply-To: <5388811E.3090802@behnel.de> References: <20140528112547.GA2814@yuyuko> <5388811E.3090802@behnel.de> Message-ID: <538886A7.8020508@behnel.de> Stefan Behnel, 30.05.2014 15:01: > Emmanuel Gil Peyrot, 28.05.2014 13:25: >> I was testing my cython codebase on top of pypy/cpyext, and I found a >> memory corruption. After investigating it with the pypy guys (thanks >> arigato!), we identified it as a cython bug: >> >> cdef void test(char *string): >> print(string) >> >> def run(array): >> test(array[0]) >> >> >> This code works fine under cpython, but looking at the generated code >> for the last line, the call to test is done with a pointer (__pyx_t_2) >> to some potentially deallocated string (__pyx_t_1), which isn?t freed >> just yet on cpython but is on pypy: >> >> __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, ?); >> __Pyx_GOTREF(__pyx_t_1); >> __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1); >> __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0; >> __pyx_f_1a_test(__pyx_t_2); >> >> >> The obvious solution is to swap the last two lines, it then works fine >> on pypy (although not necessarily if the test function stores the >> pointer somewhere, but it?s not cython?s fault then). >> >> This issue can also happen with an explicit cast: >> >> pointer = array[0] >> test(pointer) >> >> >> I?m not sure if it should be addressed the same way, because that would >> mean keeping a reference to array[0] for all the lifetime of the >> current scope, but it could still prevent obscure bugs like the memory >> corruption I had. > > Neither of these two examples should compile. Even in CPython, indexing can > not only return a safe reference but a new object, e.g. > > class Indy(object): > def __getitem__(self, i): return "abc%s" % i > > run(Indy()) > > This should crash also in CPython, or at least show unpredictable results. > > Cython has a mechanism to reject this kind of code, not sure why it > wouldn't strike here. Hmm, actually, only the cast is a problem. The function call could be made to work by explicitly keeping a reference to the object around the call. We already do this in other cases anyway, just for the coerced values. This would be a special case where we would have to store the original value instead. Stefan From stefan_ml at behnel.de Sat May 31 14:39:33 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 31 May 2014 14:39:33 +0200 Subject: [Cython] =?utf-8?q?Wrong_order_of_=5F=5FPyx=5FDECREF_when_calling?= =?utf-8?q?_a_function_with_an_implicit_str_=E2=86=92_char*_conversion=2E?= In-Reply-To: <5388811E.3090802@behnel.de> References: <20140528112547.GA2814@yuyuko> <5388811E.3090802@behnel.de> Message-ID: <5389CD85.2070804@behnel.de> Stefan Behnel, 30.05.2014 15:01: > Emmanuel Gil Peyrot, 28.05.2014 13:25: >> I was testing my cython codebase on top of pypy/cpyext, and I found a >> memory corruption. After investigating it with the pypy guys (thanks >> arigato!), we identified it as a cython bug: >> >> cdef void test(char *string): >> print(string) >> >> def run(array): >> test(array[0]) >> >> >> This code works fine under cpython, but looking at the generated code >> for the last line, the call to test is done with a pointer (__pyx_t_2) >> to some potentially deallocated string (__pyx_t_1), which isn?t freed >> just yet on cpython but is on pypy: >> >> __pyx_t_1 = __Pyx_GetItemInt(__pyx_v_array, 0, ?); >> __Pyx_GOTREF(__pyx_t_1); >> __pyx_t_2 = __Pyx_PyObject_AsString(__pyx_t_1); >> __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0; >> __pyx_f_1a_test(__pyx_t_2); >> >> >> The obvious solution is to swap the last two lines, it then works fine >> on pypy (although not necessarily if the test function stores the >> pointer somewhere, but it?s not cython?s fault then). >> >> This issue can also happen with an explicit cast: >> >> pointer = array[0] >> test(pointer) >> >> >> I?m not sure if it should be addressed the same way, because that would >> mean keeping a reference to array[0] for all the lifetime of the >> current scope, but it could still prevent obscure bugs like the memory >> corruption I had. > > Neither of these two examples should compile. Even in CPython, indexing can > not only return a safe reference but a new object, e.g. > > class Indy(object): > def __getitem__(self, i): return "abc%s" % i > > run(Indy()) > > This should crash also in CPython, or at least show unpredictable results. > > Cython has a mechanism to reject this kind of code, not sure why it > wouldn't strike here. Here's a patch. However, while writing it up, I noticed that there are many cases in CPython where these things are actually safe and nice, e.g. cdef char* charptr for charptr in list_of_byte_strings: ... CPython would allow us to avoid some reference counting here, for example (although I don't think we currently make use of it). And as long as the list doesn't change (which is the user's concern, not Cython's), this is also perfectly safe. Thus, we'd loose a nice feature, which speaks against changing the current behaviour. It's clear that at least emitting a warning in these cases would be helpful, though. An option like -Wcpython could enable special warnings on code that only works on CPython and not on other C-API implementations, such as PyPy. Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: reject_charptr_of_indexed_object.patch Type: text/x-patch Size: 1938 bytes Desc: not available URL: