From sturla.molden at gmail.com Fri Apr 3 00:38:31 2015 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 2 Apr 2015 22:38:31 +0000 (UTC) Subject: [Cython] Cython inserting unqualified module name into sys.module on python 3? References: Message-ID: <1394441294449706720.512601sturla.molden-gmail.com@news.gmane.org> Is this why I can only make scipy.spatial.cKDTree support pickle on Python 3? Since scipy.spatial.cKDTree comes from a relative import of .ckdtree.cKDTree into scipy.spatial, it somehow works on Python 3 but fails on Python 2. Sturla Nathaniel Smith wrote: > On Mar 14, 2015 2:03 PM, "Robert Bradshaw" > wrote: >> >> That is strange, looks like it was an attempt to support relative imports? >> >> https://github.com/cython/cython/blob/384cc660f5c7958524b8839ba24099fdbc6eaffd/Cython/Compiler/ModuleNode.py#L2271 > > Ah, I see. > > I don't see how this could affect relative imports, because if foo.bar > does 'import .baz', this doesn't actually trigger any access to > sys.modules["foo.bar"]. (Exception: if you have foo/__init__.pyx. Is > that actually supported?) The critical thing for relative imports is > having correct __name__ and/or __package__ module-level global > variables, and AFAICT cython is not currently doing anything to set > these up. But it probably should, because relative imports are a > thing. > > OTOH, putting the module into sys.modules *is* crucial to handle > recursive imports, i.e. where foo.pyx's module init function imports > bar.py, and bar.py imports foo. For regular python modules or for > python 2 extension modules, this works because even while foo is still > initializing, you can already get its (incomplete) module object from > sys.modules; for python 3 extension modules this won't work unless we > do it by hand. So the code that Cython is generating seems to be > correct and necessary, it just has the wrong idea about what the > fully-qualified module name is, and this breaks things. > > So I'm convinced that Cython has to know the fully-qualified module > name for correct operation, and if it's wrong then weird real bugs > will happen. Next question: how am I supposed to make this work? Maybe > I'm just missing it, but I can't find anything in the docs about how I > should tell cython that mtrand.pyx is really numpy.random.mtrand...? > > -n > >> On Sat, Mar 14, 2015 at 1:17 AM, Nathaniel Smith >> wrote: >>> Hi all, >>> >>> Can anyone shed any light on this? >>> >>> https://github.com/numpy/numpy/issues/5680 >>> >>> -n >>> >>> -- >>> Nathaniel J. Smith -- http://vorpus.org >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> https://mail.python.org/mailman/listinfo/cython-devel >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel From carlosjosepita at gmail.com Sun Apr 5 21:48:57 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 5 Apr 2015 16:48:57 -0300 Subject: [Cython] [Bug] Memoryviews in pure python mode Message-ID: Hi all, I've posted about this in the user list but after thinking about it a bit more and doing some testing, I tend to believe it's a bug. In the following code, the cython.double[:] in @cython.locals is not recognized as a type, while g() compiles fine: import cython import scipy @cython.locals(x=cython.double[:]) def f(): x = scipy.array([1,2,3], scipy.double) def g(): cdef double[:] x = scipy.array([1,2,3], scipy.double) Now, one could said memoryviews aren't supported in pure python mode (which would be a pity) but then, in the interpreter: In [48]: cy.int[:] Out[48]: int[:] In [49]: type(cy.int[:]) Out[49]: Cython.Shadow._ArrayType Shadow.py implements the machinery for the interpreter, but the compiler is not consistently supporting it. Best regards -- Carlos PS: do you know of any workaround? From carlosjosepita at gmail.com Sun Apr 5 23:37:16 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 5 Apr 2015 18:37:16 -0300 Subject: [Cython] [Bug] Memoryviews in pure python mode In-Reply-To: References: Message-ID: The problem seems to be that SliceIndexNode.analyse_as_type returns just None, thus visit_FuncDefNode throws an error in the cython.locals analysis part. Since a slice can be viewed as a type in the context of cython.locals and cython.declare, analyse_as_type shouldn't return None. This is my first contact with cython codebase so this is as far as I'm going for the time being. But with appropriate guidance I could figure out a fix, I guess. > PS: do you know of any workaround? Btw, by inspecting the code I have found a workaround using named types, that is the string representation of the type. I think the documentation for pure python mode should mention this possibility in the "C types" section, as it's a sure fallback for fancy types not directly supported as python objects. A good place to include this valuable information is after the caveat "A limited attempt is made to emulate these more complex types...". I should add that the documentation does include an example in which named types are used for signature annotations, though it's pretty elliptical. Cheers -- Carlos From carlosjosepita at gmail.com Mon Apr 6 00:49:17 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sun, 5 Apr 2015 19:49:17 -0300 Subject: [Cython] [Bug] Coercion of struct constructor nodes Message-ID: Hi all, f and g below should behave identically, shouldn't them? import cython cdef struct Point: int x int y def f(): return Point(x=10, y=10) def g(): cdef Point p = Point(x=10, y=10) return p But then f won't compile: Cannot interpret dict as type 'Python object' DictNode.coerce_to decides it can't coerce a CStructOrUnionType to a python object, but the underlying struct type does know how to coerce a struct into a dict. Here is an old post stating the same, although I don't know if it ever entered the bug tracker. https://www.mail-archive.com/cython-dev at codespeak.net/msg09121.html Best regards -- Carlos From stefan_ml at behnel.de Mon Apr 6 09:30:43 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 06 Apr 2015 09:30:43 +0200 Subject: [Cython] [Bug] Coercion of struct constructor nodes In-Reply-To: References: Message-ID: <55223623.6020008@behnel.de> Carlos Pita schrieb am 06.04.2015 um 00:49: > f and g below should behave identically, shouldn't them? > > import cython > > cdef struct Point: > int x > int y > > def f(): > return Point(x=10, y=10) > > def g(): > cdef Point p = Point(x=10, y=10) > return p Yes, they should. https://github.com/cython/cython/commit/c25a6167fbc839f2af4e0a01e40792d827492450 Stefan From carlosjosepita at gmail.com Tue Apr 7 18:39:42 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Tue, 7 Apr 2015 13:39:42 -0300 Subject: [Cython] [RFE] Add dummy compiler directive decorators for pure python mode Message-ID: Hi all, pure python mode will benefit from the addition of dummy decorators for compiler directives to Shadow.py, like in: @cython.boundscheck(False) def f(): pass AFAICS this is not currently valid inside the interpreter. Cheers -- Carlos From stefan_ml at behnel.de Tue Apr 7 20:28:54 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 07 Apr 2015 20:28:54 +0200 Subject: [Cython] [RFE] Add dummy compiler directive decorators for pure python mode In-Reply-To: References: Message-ID: <552421E6.7080307@behnel.de> Carlos Pita schrieb am 07.04.2015 um 18:39: > pure python mode will benefit from the addition of dummy decorators > for compiler directives to Shadow.py, like in: > > @cython.boundscheck(False) > def f(): pass > > AFAICS this is not currently valid inside the interpreter. Sure, why not. https://github.com/cython/cython/commit/690881089fc2bc16e44e6e88b6a4c979cd57d9ad Stefan From carlosjosepita at gmail.com Fri Apr 10 14:40:48 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Fri, 10 Apr 2015 09:40:48 -0300 Subject: [Cython] [RFE] Add dummy compiler directive decorators for pure python mode In-Reply-To: <552421E6.7080307@behnel.de> References: <552421E6.7080307@behnel.de> Message-ID: Not sure, but would it be desirable for the decorators to be less dummy and for RuntimeCompiledFunction to take the flags into account when compiling on the fly? Thank you for your prompt support. Cheers -- Carlos -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Apr 10 14:43:14 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Apr 2015 14:43:14 +0200 Subject: [Cython] [RFE] Add dummy compiler directive decorators for pure python mode In-Reply-To: References: <552421E6.7080307@behnel.de> Message-ID: <5527C562.7060108@behnel.de> Carlos Pita schrieb am 10.04.2015 um 14:40: > Not sure, but would it be desirable for the decorators to be less dummy and > for RuntimeCompiledFunction to take the flags into account when compiling > on the fly? Can you provide a pull request, including tests? Stefan From stefan_ml at behnel.de Sun Apr 12 21:41:43 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 12 Apr 2015 21:41:43 +0200 Subject: [Cython] CI tests with pypy/pypy3 Message-ID: <552ACA77.6030808@behnel.de> Hi, I set up CI tests on Jenkins for the latest pypy and pypy3 releases and invested some time into improving the situation. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pypy/ I explicitly disabled a couple of tests that either crash or otherwise won't work in pypy anyway. For example, pypy's GIL support is incomplete and inheriting from builtin types is buggy. https://github.com/cython/cython/blob/master/tests/pypy_bugs.txt It doesn't look all that bad, at least for pypy2. I consider pypy3 work in progress, but we should keep them busy with fixing bugs in cpyext. There's always room for improvements and work-arounds on our side, too, but the major issues usually come up on their side. Stefan From stefan_ml at behnel.de Fri Apr 17 07:53:26 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 17 Apr 2015 07:53:26 +0200 Subject: [Cython] PEP 489: Redesigning extension module loading Message-ID: <55309FD6.5070205@behnel.de> Hi, I've been involved in redesigning a new protocol for importing extension modules, which might make it into CPython 3.5. I think it's now ready for general consideration. http://article.gmane.org/gmane.comp.python.import/508 It solves several issues with the current scheme and brings it much closer to the behaviour of "normal" Python modules. Stefan From matthew.m.graham at gmail.com Fri Apr 17 03:09:32 2015 From: matthew.m.graham at gmail.com (Matt Graham) Date: Fri, 17 Apr 2015 02:09:32 +0100 Subject: [Cython] Possible bug: Memory leak issues when using cpython.array as memoryview and/or buffer in Python 2.7 Message-ID: Hello, I'm filing this as a bug report here as the issue tracker is closed to anonymous users. ## Summary There seems to be a memory leak issue when using the buffer and memoryview support of the cpython.array Cython interface to the inbuilt Python array.array module. ## OS / Environment Ubuntu 14.04 Cython 0.22 Python 2.7.6/2.7.9 ## Description When allocating `cpython.array` objects using the `clone` method provided in the Cython interface to a buffer or memoryview type the memory does not seem to be freed correctly when all references to the object have been removed when running Cython code with Python 2.7 (tested with both Ubuntu provided 2.7.6 version and 2.7.9 downloaded from python.org this evening). This issue does not seem to occur when running the exact same Cython code with Python 3.4.0. This appears like it could potentially be a long standing bug as the same issue was mentioned on the cython-user mailing list in November 2012 ([here][1]). Using `cpython.array` with memory views seems to be quite a typical use case: it is discussed in the documentation [here][2] and [here][3] (specifically being stated to be memory safe) and is recommended in a popular StackOverflow answer [here][4] on the best way for allocating memory for a typed memory view in Cython (based on benchmarks performed in Python 3.x). [1]: https://groups.google.com/d/msg/cython-users/CwtU_jYADgM/660O2JJuO54J [2]: http://docs.cython.org/src/userguide/memoryviews.html#cpython-array-module [3]: http://docs.cython.org/src/tutorial/array.html [4]: http://stackoverflow.com/a/21054369/4798943 ## Steps to reproduce Cython code is attached (also copied below as not sure whether attachment will be preserved in posting) to reproduce this error. Running this code with Python 2.7.9 gives output > Memory usage: 64.16015625 MiB > Starting cpython array buffer initialisation loop > Finished cpython array buffer initialisation loop > Memory usage: 459.71484375 MiB > Starting cpython array memview initialisation loop > Finished cpython array memview initialisation loop > Memory usage: 849.55078125 MiB and with Python 3.4.0 > Memory usage: 52.79296875 MiB > Starting cpython array buffer initialisation loop > Finished cpython array buffer initialisation loop > Memory usage: 52.8125 MiB > Starting cpython array memview initialisation loop > Finished cpython array memview initialisation loop > Memory usage: 52.81640625 MiB Let me know if any more details are needed. Thanks, Matt ``` # cython: boundscheck=False # cython: wraparound=False import resource import gc from cpython.array cimport array, clone def init_cpython_array_buffer(int length, int loops): cdef int i cdef array[double] arr, template = array('d') for i in range(loops): arr = clone(template, length, False) def init_cpython_array_memview(int length, int loops): cdef int i cdef double[::1] arr cdef array template = array('d') for i in range(loops): arr = clone(template, length, False) mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024. print('Memory usage: {0} MiB'.format(mem)) print('Starting cpython array buffer initialisation loop') init_cpython_array_buffer(100000, 100000) # Force a garbage collection just to be safe gc.collect() print('Finished cpython array buffer initialisation loop') mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024. print('Memory usage: {0} MiB'.format(mem)) print('Starting cpython array memview initialisation loop') init_cpython_array_memview(100000, 100000) # Force a garbage collection just to be safe gc.collect() print('Finished cpython array memview initialisation loop') mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024. print('Memory usage: {0} MiB'.format(mem)) ``` -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cpython_array_memory_leak_test.pyx Type: text/x-python Size: 1231 bytes Desc: not available URL: From stefan_ml at behnel.de Sat Apr 18 08:12:31 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 18 Apr 2015 08:12:31 +0200 Subject: [Cython] PEP 492 - Coroutines with async and await syntax Message-ID: <5531F5CF.7030605@behnel.de> Hi! Looks like there's a new PEP coming up which receives rather strong support: https://www.python.org/dev/peps/pep-0492/ It proposes new sytax constructs like async with EXPR as VAR: BLOCK async for TARGET in ITER: BLOCK that substantially simplify asynchronous code. Here's the current discussion thread on python-ideas: http://comments.gmane.org/gmane.comp.python.ideas/33009 Stefan From stefan_ml at behnel.de Tue Apr 21 07:21:36 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 21 Apr 2015 07:21:36 +0200 Subject: [Cython] Generator ABC in Py3.5 Message-ID: <5535DE60.6040503@behnel.de> Hi, I'm currently working on making the Generator protocol an ABC in Python 3.5. That will allow Cython generators to finally stand en-par with Python generators. https://bugs.python.org/issue24018 Stefan From michael at ensslin.cc Wed Apr 22 22:07:01 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Wed, 22 Apr 2015 22:07:01 +0200 Subject: [Cython] Cython produces invalid C code Message-ID: <5537FF65.4000602@ensslin.cc> Hi everybody, Cython 0.21.1, from Debian Sid, and Cython 0.22, from Gentoo, produce invalid C Code for the following .pyx file: $ cat test.pyx cimport cpython cdef extern from "test.h": cdef void foo(int i = 0) def bar(self): foo(0) $ cat test.h void foo(int i); $ cython test.pyx $ gcc -c test.c -I/usr/include/python3.4m test.c: In function ?__pyx_pf_4test_bar?: test.c:659:35: error: storage size of ?__pyx_t_1? isn?t known struct __pyx_opt_args_4test_foo __pyx_t_1; $ clang test.c -I/usr/include/python3.4m test.c:659:35: error: variable has incomplete type 'struct __pyx_opt_args_4test_foo' struct __pyx_opt_args_4test_foo __pyx_t_1; test.c:659:10: note: forward declaration of 'struct __pyx_opt_args_4test_foo' struct __pyx_opt_args_4test_foo __pyx_t_1; Note that this is a minimal example; removing anything from test.pyx fixes the issue (the 'cimport' statement, the default value for int i, and the call to foo). The issue also occurs with --cplus. Happy debugging, mic_e -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From robertwb at gmail.com Thu Apr 23 06:25:54 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Wed, 22 Apr 2015 21:25:54 -0700 Subject: [Cython] Cython produces invalid C code In-Reply-To: <5537FF65.4000602@ensslin.cc> References: <5537FF65.4000602@ensslin.cc> Message-ID: Extern functions can't have default arguments. I've made this an explicit error. On Wed, Apr 22, 2015 at 1:07 PM, Michael En?lin wrote: > Hi everybody, > > Cython 0.21.1, from Debian Sid, and Cython 0.22, from Gentoo, produce > invalid C Code for the following .pyx file: > > > > $ cat test.pyx > cimport cpython > > cdef extern from "test.h": > cdef void foo(int i = 0) > > def bar(self): > foo(0) > > > > $ cat test.h > void foo(int i); > > > > $ cython test.pyx > > > > $ gcc -c test.c -I/usr/include/python3.4m > test.c: In function ?__pyx_pf_4test_bar?: > test.c:659:35: error: storage size of ?__pyx_t_1? isn?t known > struct __pyx_opt_args_4test_foo __pyx_t_1; > > > > $ clang test.c -I/usr/include/python3.4m > test.c:659:35: error: variable has incomplete type 'struct > __pyx_opt_args_4test_foo' > struct __pyx_opt_args_4test_foo __pyx_t_1; > test.c:659:10: note: forward declaration of 'struct > __pyx_opt_args_4test_foo' > struct __pyx_opt_args_4test_foo __pyx_t_1; > > > > Note that this is a minimal example; removing anything from test.pyx > fixes the issue (the 'cimport' statement, the default value for int i, > and the call to foo). The issue also occurs with --cplus. > > > Happy debugging, > mic_e > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From michael at ensslin.cc Thu Apr 23 08:11:30 2015 From: michael at ensslin.cc (=?UTF-8?B?TWljaGFlbCBFbsOfbGlu?=) Date: Thu, 23 Apr 2015 08:11:30 +0200 Subject: [Cython] Cython produces invalid C code In-Reply-To: References: <5537FF65.4000602@ensslin.cc> Message-ID: <55388D12.3060609@ensslin.cc> Hi, On 23/04/15 06:25, Robert Bradshaw wrote: > I've made this an explicit error. thanks; I hope this will save future users some WTFs. ~mic_e -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From mouse at yandex-team.ru Thu Apr 23 11:22:37 2015 From: mouse at yandex-team.ru (Anton D. Kachalov) Date: Thu, 23 Apr 2015 12:22:37 +0300 Subject: [Cython] Dash in the executable's filename Message-ID: <39001429780957@webcorp01e.yandex-team.ru> An HTML attachment was scrubbed... URL: From robertwb at math.washington.edu Fri Apr 24 10:31:40 2015 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 24 Apr 2015 01:31:40 -0700 Subject: [Cython] Dash in the executable's filename In-Reply-To: <39001429780957@webcorp01e.yandex-team.ru> References: <39001429780957@webcorp01e.yandex-team.ru> Message-ID: Good point: https://github.com/cython/cython/commit/e0fd2b3c8265ea40084ee9981d8601bc6e11b97e On Thu, Apr 23, 2015 at 2:22 AM, Anton D. Kachalov wrote: > Hello. > > I've found that executable script with dashes in the filename lead to > produce wrong cythonized source: > > $ touch my-great-script.py > $ cython my-great-script.py --embed > $ fgrep PyInit my-great-script.c > PyMODINIT_FUNC PyInit_my-great-script(void); /*proto*/ > PyMODINIT_FUNC PyInit_my-great-script(void) > __Pyx_RefNannySetupContext("PyMODINIT_FUNC PyInit_my-great-script(void)", > 0); > m = PyInit_my-great-script(); > > So, if I don't want to import my final script elsewhere, I'm free to choose > any filename for it. > > -- > Anton D. Kachalov > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Fri Apr 24 10:45:30 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 24 Apr 2015 10:45:30 +0200 Subject: [Cython] Dash in the executable's filename In-Reply-To: References: <39001429780957@webcorp01e.yandex-team.ru> Message-ID: <553A02AA.7010805@behnel.de> Robert Bradshaw schrieb am 24.04.2015 um 10:31: > On Thu, Apr 23, 2015 at 2:22 AM, Anton D. Kachalov wrote: >> I've found that executable script with dashes in the filename lead to >> produce wrong cythonized source: >> >> $ touch my-great-script.py >> $ cython my-great-script.py --embed >> $ fgrep PyInit my-great-script.c >> PyMODINIT_FUNC PyInit_my-great-script(void); /*proto*/ >> PyMODINIT_FUNC PyInit_my-great-script(void) >> __Pyx_RefNannySetupContext("PyMODINIT_FUNC PyInit_my-great-script(void)", >> 0); >> m = PyInit_my-great-script(); >> >> So, if I don't want to import my final script elsewhere, I'm free to choose >> any filename for it. > > Good point: https://github.com/cython/cython/commit/e0fd2b3c8265ea40084ee9981d8601bc6e11b97e I think the emphasis is on "*if* I don't want to import ...". Just because you want to be able to run a module as a program doesn't mean you will never import it as module, does it? I think being able to get both from the same generated .c file isn't a bad thing. Stefan From robertwb at gmail.com Fri Apr 24 18:36:52 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 24 Apr 2015 09:36:52 -0700 Subject: [Cython] Dash in the executable's filename In-Reply-To: <553A02AA.7010805@behnel.de> References: <39001429780957@webcorp01e.yandex-team.ru> <553A02AA.7010805@behnel.de> Message-ID: There are other differences when complaining in embed mode, such as generating a main method. I view the c file as an intermediate that depends on the compilation mode--if you want both generate two .c files. On Fri, Apr 24, 2015 at 1:45 AM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 24.04.2015 um 10:31: >> On Thu, Apr 23, 2015 at 2:22 AM, Anton D. Kachalov wrote: >>> I've found that executable script with dashes in the filename lead to >>> produce wrong cythonized source: >>> >>> $ touch my-great-script.py >>> $ cython my-great-script.py --embed >>> $ fgrep PyInit my-great-script.c >>> PyMODINIT_FUNC PyInit_my-great-script(void); /*proto*/ >>> PyMODINIT_FUNC PyInit_my-great-script(void) >>> __Pyx_RefNannySetupContext("PyMODINIT_FUNC PyInit_my-great-script(void)", >>> 0); >>> m = PyInit_my-great-script(); >>> >>> So, if I don't want to import my final script elsewhere, I'm free to choose >>> any filename for it. >> >> Good point: https://github.com/cython/cython/commit/e0fd2b3c8265ea40084ee9981d8601bc6e11b97e > > I think the emphasis is on "*if* I don't want to import ...". Just because > you want to be able to run a module as a program doesn't mean you will > never import it as module, does it? I think being able to get both from the > same generated .c file isn't a bad thing. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Sat Apr 25 17:40:19 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 25 Apr 2015 17:40:19 +0200 Subject: [Cython] release preparations for 0.22.1 and 0.23 Message-ID: <553BB563.5050100@behnel.de> Hi, I think it's about time for a new release. I propose to relase 0.22.1 in the next days, and then start the release cycle for 0.23. Please make sure everything that you consider a safe bug fix is in the 0.22.x branch. I already copied Jeroen's latest fixes over. Stefan From jdemeyer at cage.ugent.be Sat Apr 25 23:17:44 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Sat, 25 Apr 2015 23:17:44 +0200 Subject: [Cython] release preparations for 0.22.1 and 0.23 In-Reply-To: <553BB563.5050100@behnel.de> References: <553BB563.5050100@behnel.de> Message-ID: <553C0478.9030400@cage.ugent.be> On 2015-04-25 17:40, Stefan Behnel wrote: > Hi, > > I think it's about time for a new release. I propose to relase 0.22.1 in > the next days, and then start the release cycle for 0.23. > > Please make sure everything that you consider a safe bug fix is in the > 0.22.x branch. Does it include the memleak fix? That would be good to have. From stefan_ml at behnel.de Sun Apr 26 08:23:00 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 26 Apr 2015 08:23:00 +0200 Subject: [Cython] release preparations for 0.22.1 and 0.23 In-Reply-To: <553C0478.9030400@cage.ugent.be> References: <553BB563.5050100@behnel.de> <553C0478.9030400@cage.ugent.be> Message-ID: <553C8444.30607@behnel.de> Jeroen Demeyer schrieb am 25.04.2015 um 23:17: > On 2015-04-25 17:40, Stefan Behnel wrote: >> Please make sure everything that you consider a safe bug fix is in the >> 0.22.x branch. > Does it include the memleak fix? That would be good to have. Good catch. It does now. :) Stefan From stefan_ml at behnel.de Sun Apr 26 20:16:04 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 26 Apr 2015 20:16:04 +0200 Subject: [Cython] Dash in the executable's filename In-Reply-To: References: <39001429780957@webcorp01e.yandex-team.ru> Message-ID: <553D2B64.7080004@behnel.de> Robert Bradshaw schrieb am 24.04.2015 um 10:31: > On Thu, Apr 23, 2015 at 2:22 AM, Anton D. Kachalov wrote: >> I've found that executable script with dashes in the filename lead to >> produce wrong cythonized source: >> >> $ touch my-great-script.py >> $ cython my-great-script.py --embed >> $ fgrep PyInit my-great-script.c >> PyMODINIT_FUNC PyInit_my-great-script(void); /*proto*/ >> PyMODINIT_FUNC PyInit_my-great-script(void) >> __Pyx_RefNannySetupContext("PyMODINIT_FUNC PyInit_my-great-script(void)", >> 0); >> m = PyInit_my-great-script(); >> >> So, if I don't want to import my final script elsewhere, I'm free to choose >> any filename for it. > > Good point: https://github.com/cython/cython/commit/e0fd2b3c8265ea40084ee9981d8601bc6e11b97e This change prevents finding the external .pxd file for the module. I noticed this by pure luck when several of the Cython optimised benchmarks in Demos/benchmarks/ suddenly dropped substantially in performance. https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-cybenchmarks-py3k/1400/artifact/bench_chart.html The build job compiles them as Python-embedded programs. Stefan From robertwb at gmail.com Mon Apr 27 18:15:03 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 27 Apr 2015 09:15:03 -0700 Subject: [Cython] Dash in the executable's filename In-Reply-To: <553D2B64.7080004@behnel.de> References: <39001429780957@webcorp01e.yandex-team.ru> <553D2B64.7080004@behnel.de> Message-ID: On Sun, Apr 26, 2015 at 11:16 AM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 24.04.2015 um 10:31: >> On Thu, Apr 23, 2015 at 2:22 AM, Anton D. Kachalov wrote: >>> I've found that executable script with dashes in the filename lead to >>> produce wrong cythonized source: >>> >>> $ touch my-great-script.py >>> $ cython my-great-script.py --embed >>> $ fgrep PyInit my-great-script.c >>> PyMODINIT_FUNC PyInit_my-great-script(void); /*proto*/ >>> PyMODINIT_FUNC PyInit_my-great-script(void) >>> __Pyx_RefNannySetupContext("PyMODINIT_FUNC PyInit_my-great-script(void)", >>> 0); >>> m = PyInit_my-great-script(); >>> >>> So, if I don't want to import my final script elsewhere, I'm free to choose >>> any filename for it. >> >> Good point: https://github.com/cython/cython/commit/e0fd2b3c8265ea40084ee9981d8601bc6e11b97e > > This change prevents finding the external .pxd file for the module. I > noticed this by pure luck when several of the Cython optimised benchmarks > in Demos/benchmarks/ suddenly dropped substantially in performance. > > https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-cybenchmarks-py3k/1400/artifact/bench_chart.html > > The build job compiles them as Python-embedded programs. Arguably that's still correct behavior, similarly to how one can do "import foo" when executing "python foo.py" and it's a new module. However, I've undone this change for now (though "cython my-great-script.py --embed" is now an error rather than produce bad C code). From robertwb at gmail.com Mon Apr 27 20:15:28 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 27 Apr 2015 11:15:28 -0700 Subject: [Cython] New hosting Message-ID: Since Cython's inception, we've been able to take advantage of William Stein's and University of Washington's infrastructure for hosting the Cython project along side that of Sage. However, due to UW policy, those days are coming to and end. Maybe it's time--Cython has come a long way from its birth of SageX + lxml. So the question is where to move. Our primary needs are not that large: we've got a site, a bugtracker (trac), and a continuous build (jenkins) currently being served (the source code and wiki have already been migrated to github). I would propose that we look into moving everything we can over to github. For starters, they now offer serving simple sites from a repository (cython.org) so that seems an obvious choice. I know their bug tracking v1 was considered far inferior to trac, but the situation may be better now (at least worth exploring). We also have travis.ci, which isn't as configurable as jenkins, but may be good enough. (The biggest deficiency is that it probably wouldn't allow for building and testing Sage regularly, or benchmarks, this is the one thing that we may need to find/provide custom hosting for.) Thoughts? From stefan_ml at behnel.de Mon Apr 27 20:56:09 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Apr 2015 20:56:09 +0200 Subject: [Cython] New hosting In-Reply-To: References: Message-ID: <553E8649.4000700@behnel.de> Robert Bradshaw schrieb am 27.04.2015 um 20:15: > We also have travis.ci, > which isn't as configurable as jenkins, but may be good enough. (The > biggest deficiency is that it probably wouldn't allow for building and > testing Sage regularly, or benchmarks, this is the one thing that we > may need to find/provide custom hosting for.) ... and it won't easily allow us to track the CPython development branches and build against them. A really neat feature of our Jenkins server was that it could nicely store away pre-built artefacts for later reuse on the same machine, such as a series of CPython branch builds, and just happily rebuilt them regularly on external changes. That totally helped spotting integration problems as early as possible, including bugs on both sides. Stefan From matthew.brett at gmail.com Mon Apr 27 22:59:11 2015 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 27 Apr 2015 13:59:11 -0700 Subject: [Cython] New hosting In-Reply-To: <553E8649.4000700@behnel.de> References: <553E8649.4000700@behnel.de> Message-ID: Hi, On Mon, Apr 27, 2015 at 11:56 AM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 27.04.2015 um 20:15: >> We also have travis.ci, >> which isn't as configurable as jenkins, but may be good enough. (The >> biggest deficiency is that it probably wouldn't allow for building and >> testing Sage regularly, or benchmarks, this is the one thing that we >> may need to find/provide custom hosting for.) > > ... and it won't easily allow us to track the CPython development branches > and build against them. A really neat feature of our Jenkins server was > that it could nicely store away pre-built artefacts for later reuse on the > same machine, such as a series of CPython branch builds, and just happily > rebuilt them regularly on external changes. That totally helped spotting > integration problems as early as possible, including bugs on both sides. Some of us use scikit-learn's rackspace account for that. I'm sure if you ask Olivier Grisel nicely, from scikit-learn - he'd let you use it too. We use travic-ci encrypted tokens to push artefacts like wheels up to the rackspace CDN, and then pull them down again for testing. Can point you to relevant stuff if you are interested... Cheers, Matthew From robertwb at gmail.com Tue Apr 28 08:08:53 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 27 Apr 2015 23:08:53 -0700 Subject: [Cython] New hosting In-Reply-To: References: <553E8649.4000700@behnel.de> Message-ID: On Mon, Apr 27, 2015 at 1:59 PM, Matthew Brett wrote: > Hi, > > On Mon, Apr 27, 2015 at 11:56 AM, Stefan Behnel wrote: >> Robert Bradshaw schrieb am 27.04.2015 um 20:15: >>> We also have travis.ci, >>> which isn't as configurable as jenkins, but may be good enough. (The >>> biggest deficiency is that it probably wouldn't allow for building and >>> testing Sage regularly, or benchmarks, this is the one thing that we >>> may need to find/provide custom hosting for.) >> >> ... and it won't easily allow us to track the CPython development branches >> and build against them. A really neat feature of our Jenkins server was >> that it could nicely store away pre-built artefacts for later reuse on the >> same machine, such as a series of CPython branch builds, and just happily >> rebuilt them regularly on external changes. That totally helped spotting >> integration problems as early as possible, including bugs on both sides. That's a good point. > Some of us use scikit-learn's rackspace account for that. I'm sure if > you ask Olivier Grisel nicely, from scikit-learn - he'd let you use it > too. > > We use travic-ci encrypted tokens to push artefacts like wheels up to > the rackspace CDN, and then pull them down again for testing. > > Can point you to relevant stuff if you are interested... If we go down this route, for sure. From linkmauve at linkmauve.fr Wed Apr 29 20:11:28 2015 From: linkmauve at linkmauve.fr (Emmanuel Gil Peyrot) Date: Wed, 29 Apr 2015 20:11:28 +0200 Subject: [Cython] [PATCH 1/2] Move ~/.pyxbld to $XDG_CACHE_HOME/pyxbld Message-ID: <1430331089-15169-1-git-send-email-linkmauve@linkmauve.fr> --- pyximport/pyximport.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/pyximport/pyximport.py b/pyximport/pyximport.py index 4fd7fe9..710c5eb 100644 --- a/pyximport/pyximport.py +++ b/pyximport/pyximport.py @@ -466,9 +466,10 @@ def install(pyximport=True, pyimport=False, build_dir=None, build_in_temp=True, will not work for most .py files, and will therefore only slow down your imports. Use at your own risk. - By default, compiled modules will end up in a ``.pyxbld`` - directory in the user's home directory. Passing a different path - as ``build_dir`` will override this. + By default, compiled modules will end up in a ``pyxbld`` directory + in the directory pointed by the ``XDG_CACHE_HOME`` environment + variable, or in ``~/.cache/pyxbld`` if ``XDG_CACHE_HOME`` isn?t + set. Passing a different path as ``build_dir`` will override this. ``build_in_temp=False`` will produce the C files locally. Working with complex dependencies and debugging becomes more easy. This @@ -501,8 +502,9 @@ def install(pyximport=True, pyimport=False, build_dir=None, build_in_temp=True, runtime for .py files and Py2 for .pyx files. """ if not build_dir: - build_dir = os.path.join(os.path.expanduser('~'), '.pyxbld') - + build_dir = os.path.join(os.environ.get('XDG_CACHE_HOME', + os.path.join(os.path.expanduser('~'), '.cache')), 'pyxbld') + global pyxargs pyxargs = PyxArgs() #$pycheck_no pyxargs.build_dir = build_dir -- 2.3.5 From linkmauve at linkmauve.fr Wed Apr 29 20:11:29 2015 From: linkmauve at linkmauve.fr (Emmanuel Gil Peyrot) Date: Wed, 29 Apr 2015 20:11:29 +0200 Subject: [Cython] =?utf-8?q?=5BPATCH_2/2=5D_Make_the_Cython_cache_director?= =?utf-8?q?y_fallback_to_=7E/=2Ecache/cython_if_=24XDG=5FCACHE=5FHO?= =?utf-8?q?ME_isn=E2=80=99t_set?= In-Reply-To: <1430331089-15169-1-git-send-email-linkmauve@linkmauve.fr> References: <1430331089-15169-1-git-send-email-linkmauve@linkmauve.fr> Message-ID: <1430331089-15169-2-git-send-email-linkmauve@linkmauve.fr> --- Cython/Utils.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/Cython/Utils.py b/Cython/Utils.py index 22f6db7..563989f 100644 --- a/Cython/Utils.py +++ b/Cython/Utils.py @@ -327,7 +327,8 @@ def get_cython_cache_dir(): 1. CYTHON_CACHE_DIR 2. (OS X): ~/Library/Caches/Cython (posix not OS X): XDG_CACHE_HOME/cython if XDG_CACHE_HOME defined - 3. ~/.cython + 3. ~/.cache/cython + 4. ~/.cython """ if 'CYTHON_CACHE_DIR' in os.environ: @@ -338,8 +339,8 @@ def get_cython_cache_dir(): if sys.platform == 'darwin': parent = os.path.expanduser('~/Library/Caches') else: - # this could fallback on ~/.cache - parent = os.environ.get('XDG_CACHE_HOME') + parent = os.environ.get('XDG_CACHE_HOME', + os.path.join(os.path.expanduser('~'), '.cache')) if parent and os.path.isdir(parent): return os.path.join(parent, 'cython') -- 2.3.5 From linkmauve at linkmauve.fr Wed Apr 29 20:46:26 2015 From: linkmauve at linkmauve.fr (Emmanuel Gil Peyrot) Date: Wed, 29 Apr 2015 20:46:26 +0200 Subject: [Cython] [PATCH 1/2] Move ~/.pyxbld to $XDG_CACHE_HOME/pyxbld In-Reply-To: <1430331089-15169-1-git-send-email-linkmauve@linkmauve.fr> References: <1430331089-15169-1-git-send-email-linkmauve@linkmauve.fr> Message-ID: <20150429184626.GA12992@yuyuko> Actually, I think it?d be better to move the pyxbld directory directly under the cython cache directory, as it is part of cython itself. Here is an attached patch updated with this behaviour. -- Emmanuel Gil Peyrot -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-Move-.pyxbld-to-XDG_CACHE_HOME-cython-pyxbld.patch Type: text/x-diff Size: 1680 bytes Desc: not available URL: