From frank.schlimbach at intel.com Mon Jan 6 06:09:55 2020 From: frank.schlimbach at intel.com (Schlimbach, Frank) Date: Mon, 6 Jan 2020 11:09:55 +0000 Subject: [Cython] Adding GPU support to cython Message-ID: Hello, I would like to work on extending cython with a way to offload cython code to a GPU. I found to related CEPs (https://github.com/cython/cython/wiki/enhancements-opencl and https://github.com/cython/cython/wiki/enchancements-metadefintions). My current thinking is that a solution along the OpenCL CEP is most effective, it does require many code changes and seems to provide a good tradeoff between usability and efficiency. I would like to suggest a few modifications to this approach, like * using SYCL instead of OpenCL to closely follow existing parallel/prange semantics more easily * selecting the device (CPU, GPU) per region rather than per file * maybe allowing calling appropriately annotated and written external functions I would be very grateful for any thoughts about this topic in general and for any advice on how to approach this so that a solution is found that is most broadly useful and most cythonic. Cheers frank Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabiofz at gmail.com Mon Jan 6 07:20:09 2020 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Mon, 6 Jan 2020 09:20:09 -0300 Subject: [Cython] AST from Cython files Message-ID: Hi all, I'm currently using: from Cython.Compiler.TreeFragment import parse_from_strings mod = parse_from_strings(name, source) to obtain the AST and it seems to be working well for me (I later convert that AST to my own AST), but I'm currently using this for an IDE, so, I'd like to be able to obtain the AST even if there are errors in the source code. So, I was thinking about making a pull request which would allow Cython to provide the AST and recover from errors (it'd return the AST and a list of errors instead of raising the first error in this API). So, I'd like to now from cython devs, before providing a pull request for that if such a pull request would be Ok or if you see some issue in this change. Thanks, Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From omcsesz at gmail.com Wed Jan 8 16:47:24 2020 From: omcsesz at gmail.com (=?UTF-8?B?b23DoXIgc3dlaWTDoW4=?=) Date: Wed, 8 Jan 2020 22:47:24 +0100 Subject: [Cython] Fwd: .gitignore file in github repo In-Reply-To: References: Message-ID: Hello, All! Is there a way for me to contribute to the C files in the Cython folder? When I changed them, it does not show in the changed files list (e.g. in Git GUI). Why are the C files put in the gitignore file? Omar Sweidan -------------- next part -------------- An HTML attachment was scrubbed... URL: From newt0311 at gmail.com Fri Jan 10 05:22:43 2020 From: newt0311 at gmail.com (Prakhar Goel) Date: Fri, 10 Jan 2020 05:22:43 -0500 Subject: [Cython] Fwd: .gitignore file in github repo In-Reply-To: References: Message-ID: I assume that was done to avoid including all the C files that the Cython compiler generates. You can specifically tell git not to ignore your C file using a ! pattern in the .gitignore file. More details here: https://git-scm.com/docs/gitignore. -- ________________________ Warm Regards Prakhar Goel From stefan_ml at behnel.de Fri Jan 10 05:27:07 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Jan 2020 11:27:07 +0100 Subject: [Cython] Fwd: .gitignore file in github repo In-Reply-To: References: Message-ID: <579fe27a-df6c-ad05-516a-47e19417413b@behnel.de> om?r sweid?n schrieb am 08.01.20 um 22:47: > Is there a way for me to contribute to the C files in the Cython folder? > When I changed them, it does not show in the changed files list (e.g. in > Git GUI). Why are the C files put in the gitignore file? We exclude all files that are generated by Cython. There is no use in changing them, and they will be overwritten by a rebuild. Change the Python files instead that they are generated from. Stefan From omcsesz at gmail.com Fri Jan 10 05:28:08 2020 From: omcsesz at gmail.com (=?UTF-8?B?b23DoXIgc3dlaWTDoW4=?=) Date: Fri, 10 Jan 2020 11:28:08 +0100 Subject: [Cython] Fwd: .gitignore file in github repo In-Reply-To: <579fe27a-df6c-ad05-516a-47e19417413b@behnel.de> References: <579fe27a-df6c-ad05-516a-47e19417413b@behnel.de> Message-ID: I'm sorry, I thought they aren't generated. On Fri, Jan 10, 2020 at 11:27 AM Stefan Behnel wrote: > om?r sweid?n schrieb am 08.01.20 um 22:47: > > Is there a way for me to contribute to the C files in the Cython folder? > > When I changed them, it does not show in the changed files list (e.g. in > > Git GUI). Why are the C files put in the gitignore file? > > We exclude all files that are generated by Cython. There is no use in > changing them, and they will be overwritten by a rebuild. Change the Python > files instead that they are generated from. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Jan 24 05:22:18 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 24 Jan 2020 11:22:18 +0100 Subject: [Cython] Adding GPU support to cython In-Reply-To: References: Message-ID: Hi Frank, sorry for leaving this unresponded for a while. I'm far from an expert in this, but it looks like no-one else jumped in, so here's my response. Schlimbach, Frank schrieb am 06.01.20 um 12:09: > I would like to work on extending cython with a way to offload cython code to a GPU. I found to related CEPs (https://github.com/cython/cython/wiki/enhancements-opencl and https://github.com/cython/cython/wiki/enchancements-metadefintions). So, just for a bit of context, these CEPs were written a decade ago, and Cython's code base, feature set, and the general Python ecosystem have evolved a lot since then. For example, "cython.parallel" and "prange()" weren't even implemented back then. > My current thinking is that a solution along the OpenCL CEP is most effective, it does require many code changes and seems to provide a good tradeoff between usability and efficiency. > > I would like to suggest a few modifications to this approach, like > > * using SYCL instead of OpenCL to closely follow existing parallel/prange semantics more easily > * selecting the device (CPU, GPU) per region rather than per file > * maybe allowing calling appropriately annotated and written external functions > > I would be very grateful for any thoughts about this topic in general and for any advice on how to approach this so that a solution is found that is most broadly useful and most cythonic. It would definitely be cool to generate GPU support from the existing Cython patterns, in addition to the OpenMP code that we already generate. If that can be done, then users could enable GPU support by adding a C compiler define to their CFLAGS (rather than rerunning Cython), or even select between the two versions at runtime. If the GPU support is per region, then how is the code section shipped to the GPU? Is the infrastructure for this provided by the OpenCL framework or does the user or the module need to set something up in addition? Finally, generally speaking: - PR welcome - simple approach preferred (at least to get this started and prototyped) - discussion welcome on this mailing list - GitHub feature ticket seems to be missing, with a link to the ML thread https://mail.python.org/pipermail/cython-devel/2020-January/005262.html Stefan From frank.schlimbach at intel.com Fri Jan 24 06:55:11 2020 From: frank.schlimbach at intel.com (Schlimbach, Frank) Date: Fri, 24 Jan 2020 11:55:11 +0000 Subject: [Cython] Adding GPU support to cython In-Reply-To: References: Message-ID: Hi Stefan, thanks for your response. Good to hear this is still of interest. Yes, I realized these are rather old CEPs. I spent some time with looking into the Cython code and concluded that it'd be the most consistent (and simplest) approach to stick with OpenMP and use it's offload pragmas (e.g. 'target' introduced in 4.5). Given a properly setup compiler this would in theory only require one or two compiler flags to enable offloading. I even have a first prototype which generates code that existing compilers seem to swallow. It's not ready for a PR since I have not been able to get it linked an run on GPU and I wanted to get some general feedback first. You can find the code on my offload branch https://github.com/fschlimb/cython/tree/offload (it's wip so please apologize that not all comments have been updated yet to reflect my changes). Here's what it does: - accept a new 'with' directive 'device' which marks a region/block to be offloaded to a device (OpenMP target) - I also considered extending 'gil' or 'parallel' to accept an optional 'device' argument but an extra directive seemed more general/flexible to also allow non-parallel code - I don't believe we should try to automate offloading right now. Once we have something that works on explicit demand we can still think about a performance model and auto-enable offloading. - the DeviceWithBlockNode is added to the 'parallel stack' and can occur only as the outmost parallel directive - a 'with device()' requires 'nogil' - a 'with device()' will create a new scope annotated with a '#pragma omp target' - all variables which get assigned within the 'with device()' block are currently mapped as 'tofrom' - all other variables used are mapped as 'to' - identifying 'from' candidates is harder and not yet done (need to know that there is required allocation but no assignment before the 'with device()' block) - identifying 'alloc' candidates would also need additional analysis (e.g. not used outside the 'device()' block) - all object mode stuff (like exceptions for error handling) are currently disabled in a 'with device()' block Example: def f(int[:,::1] X): cdef int v = 1 cdef int i with gil, device(), parallel(): for i in prange(4): X[i] = v the 'with device' block becomes something like (simplified) { size_t __pyx_v_X__count = __pyx_v_X.shape[0]*__pyx_v_X.shape[1]; #pragma omp target map(to: __pyx_v_v) map(tofrom: __pyx_v_i , __pyx_v_X.data[0:__pyx_v_X__count], __pyx_v_X.memview, __pyx_v_X.shape, __pyx_v_X.strides, __pyx_v_X.suboffsets) { #pragma omp parallel #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) for((__pyx_v_i=0; __pyx_v_i<4; ++__pyx_v_i) { __pyx_v_X[__pyx_v_i] = __pyx_v_v; } } } There are lots of things to be added and improved, in particular I am currently adding an optional argument 'map' to 'device()' which allows manually setting the map-clauses for each variable. This is necessary to allow not only optimizations but also sending only partial array data to/from the device (like when the device memory cannot hold an entire array the developer would block the computation). We can probably add some magic for simple cases but there is probably no solution for the general problem of determining the accessed index-space. Among others, things to also look at include - non-contiguous arrays/memviews - overlapping arrays/memviews - keeping data on the device between 'with device()' blocks (USM (unified shared memory) or omp target data?) - error handling - tests - docu/comments I found that the functionality I needed to touch is somewhat scattered around the compiler pipeline. It might be worth thinking about restructuring a few things to make the whole OpenMP/parallel/offload stuff more maintainable. Of course you might see other solutions than mine which make this simpler. Any thoughts/feedback/usecases appreciated frank -----Original Message----- From: cython-devel On Behalf Of Stefan Behnel Sent: Friday, January 24, 2020 11:22 AM To: cython-devel at python.org Subject: Re: [Cython] Adding GPU support to cython Hi Frank, sorry for leaving this unresponded for a while. I'm far from an expert in this, but it looks like no-one else jumped in, so here's my response. Schlimbach, Frank schrieb am 06.01.20 um 12:09: > I would like to work on extending cython with a way to offload cython code to a GPU. I found to related CEPs (https://github.com/cython/cython/wiki/enhancements-opencl and https://github.com/cython/cython/wiki/enchancements-metadefintions). So, just for a bit of context, these CEPs were written a decade ago, and Cython's code base, feature set, and the general Python ecosystem have evolved a lot since then. For example, "cython.parallel" and "prange()" weren't even implemented back then. > My current thinking is that a solution along the OpenCL CEP is most effective, it does require many code changes and seems to provide a good tradeoff between usability and efficiency. > > I would like to suggest a few modifications to this approach, like > > * using SYCL instead of OpenCL to closely follow existing parallel/prange semantics more easily > * selecting the device (CPU, GPU) per region rather than per file > * maybe allowing calling appropriately annotated and written external functions > > I would be very grateful for any thoughts about this topic in general and for any advice on how to approach this so that a solution is found that is most broadly useful and most cythonic. It would definitely be cool to generate GPU support from the existing Cython patterns, in addition to the OpenMP code that we already generate. If that can be done, then users could enable GPU support by adding a C compiler define to their CFLAGS (rather than rerunning Cython), or even select between the two versions at runtime. If the GPU support is per region, then how is the code section shipped to the GPU? Is the infrastructure for this provided by the OpenCL framework or does the user or the module need to set something up in addition? Finally, generally speaking: - PR welcome - simple approach preferred (at least to get this started and prototyped) - discussion welcome on this mailing list - GitHub feature ticket seems to be missing, with a link to the ML thread https://mail.python.org/pipermail/cython-devel/2020-January/005262.html Stefan _______________________________________________ cython-devel mailing list cython-devel at python.org https://mail.python.org/mailman/listinfo/cython-devel Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 From skaller at internode.on.net Mon Jan 27 00:56:00 2020 From: skaller at internode.on.net (John Skaller2) Date: Mon, 27 Jan 2020 16:56:00 +1100 Subject: [Cython] Hello Message-ID: Hi! I have just built Cython but haven?t used it yet. I want to check I understand what it does whilst examing the sources in the repository. Please let me know if I have it wrong! Given a Python file, Cython parses it, and translates it to the equivalent C, which can then be compiled to binary, thereby bypassing the overhead of interpreting bytecode, but still executing the same accesses to the CPython run time API that the interpreter would. So there may be a small speed improvement, or maybe not (because it misses optimisations the interpreter might be able to spot). The binary will typically be a C extension module that can be loaded and operate the same way as the original Python. Now, Cython is an extension of Python which allows some extra stuff, including type annotations, and other directives related to integration with C. These can be used to facilitate integration with external C libraries directly, and mapping into Python, as if the code were written in C, only we?re using a Python like language representing a subset of C instead of C. Additionally, the compiler recognises the type annotations, and can reduce or eliminate run time type checks, improving performance, or even replacing common constructions in Python which much faster ones that do the same job ?closer to the metal?. To make this work, the CPython API itself is represented in a set of *.pxd files found in the repository in Includes/cpython, splitting the logic of the compiler roughly into two parts: the front and back end. The front end groks Python and Cython code whilst the back end generates the actual C. === Just FYI, I?m the developer of a programming language, Felix, which is C++ code generator. You can think of it as a meta-programming language for C++ with a proper type system. Felix binds C/C++ code with statements like: type PyObject = ?PyObject*?; fun add: PyObject * PyObject -> PyObject = ?Py_AddLong($1)?; and can use the bindings like: var a : PyObject = ?. var b: PyObject = ... var sum = add (a,b); so in some ways its doing the same kind of job as Cython, except it isn?t specialised to bind to Python, it can bind to anything written in C or C++. Including the Python API as illustrated. Right now, I?ve writing a program to translate Include/python/*.pxd files to Felix bindings, to save typing everything by hand. Its even possible I can teach my compiler to read *.pxd files directly, which would make Cython a sublanguage of Felix. Felix is very good at defining DSSLs, Domain Specific Sub Languages, the grammar of the language is defined in the standard library. In the process of doing this I may find some issues in Cython which I?ll report. Hope that helps. As I learn more I may be able to contribute to the project directly. One possible future goal is to replace NumPy with something much better. I also have code written in Python that I might translate to C using Cython. It would be kind of interesting to use Cython to generate C, and then create bindings to that C in Felix, so instead of calling the Python C API, we call the Cython generated API instead, allowing people to write libraries for Felix in Cython instead of C or Felix. However that?s a more major integration task. You?d want Cython to generate the Felix bindings, or at least output meta-data that would allow them to be generated easily .. such as .. a *.pxd file !! ? John Skaller skaller at internode.on.net From greg.ewing at canterbury.ac.nz Mon Jan 27 04:23:03 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 27 Jan 2020 22:23:03 +1300 Subject: [Cython] Hello In-Reply-To: References: Message-ID: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> On 27/01/20 6:56 pm, John Skaller2 wrote: > Felix binds C/C++ code with statements like: > > type PyObject = ?PyObject*?; > fun add: PyObject * PyObject -> PyObject = ?Py_AddLong($1)?; > > and can use the bindings like: > > var a : PyObject = ?. > var b: PyObject = ... > var sum = add (a,b); How does it deal with reference counting and exception handling? -- Greg From stefan_ml at behnel.de Mon Jan 27 08:02:04 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Jan 2020 14:02:04 +0100 Subject: [Cython] Hello In-Reply-To: References: Message-ID: <296c23dd-96e3-2eaf-c589-466d1a4a962b@behnel.de> John Skaller2 schrieb am 27.01.20 um 06:56: > Hi! I have just built Cython but haven?t used it yet. > > I want to check I understand what it does whilst examing the sources in the repository. > Please let me know if I have it wrong! > > Given a Python file, Cython parses it, and translates it to the equivalent C, which can > then be compiled to binary, thereby bypassing the overhead of interpreting bytecode, > but still executing the same accesses to the CPython run time API that the interpreter would. Not the same, just equivalent operations. Cython often generates code that bypasses the "usual" C-API calls. > So there may be a small speed improvement, or maybe not (because it misses optimisations > the interpreter might be able to spot). Yes, it goes both ways. Cython generates many fast-paths that the interpreter doesn't have or cannot easily provide, so there is a high chance that compiled code is faster than interpreted code. The speed improvement is usually somewhere around 30% without type annotations, but highly dependant on your actual code. OTOH, recent CPython versions added some internal infrastructure that speeds up important operations, which isn't always easy to use from external tools like Cython. So there are a few cases where the runtime has an inherent advantage, although it doesn't always show much in comparison. And we keep fighting back. :) > The binary will typically be a C extension module > that can be loaded and operate the same way as the original Python. > > Now, Cython is an extension of Python which allows some extra stuff, including > type annotations, and other directives related to integration with C. These can be > used to facilitate integration with external C libraries directly, and mapping into > Python, as if the code were written in C, only we?re using a Python like language > representing a subset of C instead of C. It's really mostly about data types (and Cython's mixed Python/C type system). Python allows you to do a lot of seemingly different things in the same syntactic constructs, and Cython does the same when it generates the C code by adapting it to the data types that the source operates on. > Additionally, the compiler recognises the type annotations, and can reduce or > eliminate run time type checks, improving performance, or even replacing > common constructions in Python which much faster ones that do the same job > ?closer to the metal?. Yes. > To make this work, the CPython API itself is represented in a set of *.pxd files > found in the repository in Includes/cpython No. :) These files are only for end users to allow importing pre-declared parts of the CPython C-API for their own use (in case they feel like it). Cython itself does not use or need them, but if they help you? > splitting the logic of the compiler > roughly into two parts: the front and back end. The front end groks Python > and Cython code whilst the back end generates the actual C. Back to a Yes. > Just FYI, I?m the developer of a programming language, Felix, which is C++ code > generator. You can think of it as a meta-programming language for C++ with a > proper type system. Felix binds C/C++ code with statements like: > > type PyObject = ?PyObject*?; > fun add: PyObject * PyObject -> PyObject = ?Py_AddLong($1)?; > > and can use the bindings like: > > var a : PyObject = ?. > var b: PyObject = ... > var sum = add (a,b); > > so in some ways its doing the same kind of job as Cython, except it isn?t > specialised to bind to Python, it can bind to anything written in C or C++. > Including the Python API as illustrated. As Greg's question hinted, Cython knows a lot about the object reference counting that CPython uses for garbage collection. That's one of the main reasons why people prefer it over using the CPython C-API directly. Writing correct code in the latter is quite difficult and requires a lot of discipline. Cython gives you that for free. > One possible future goal is to replace NumPy with something much better. You're not the first. :) Look at other projects like Pythran, Numba, Theano, numexpr, ? Pythran actually integrates with Cython's type system to generate C++ code from NumPy expressions. > I also have code written in Python that I might translate to C using Cython. > It would be kind of interesting to use Cython to generate C, and then create > bindings to that C in Felix, so instead of calling the Python C API, we call > the Cython generated API instead, allowing people to write libraries for > Felix in Cython instead of C or Felix. However that?s a more major integration task. > You?d want Cython to generate the Felix bindings, or at least output meta-data > that would allow them to be generated easily .. such as .. a *.pxd file !! I would encourage you to generate Cython code instead of C/C++ directly, if you want to interact with Python. There's no need to create yet another "generator for C-API calls that isn't as good as Cython". Even if you need to mix in C/C++ code, Cython will allow you to do that, e.g. via verbatim code sections. http://docs.cython.org/en/latest/src/userguide/external_C_code.html#including-verbatim-c-code Stefan From skaller at internode.on.net Mon Jan 27 20:07:09 2020 From: skaller at internode.on.net (John Skaller2) Date: Tue, 28 Jan 2020 12:07:09 +1100 Subject: [Cython] Hello In-Reply-To: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> Message-ID: <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> > On 27 Jan 2020, at 20:23, Greg Ewing wrote: > > On 27/01/20 6:56 pm, John Skaller2 wrote: >> Felix binds C/C++ code with statements like: >> type PyObject = ?PyObject*?; >> fun add: PyObject * PyObject -> PyObject = ?Py_AddLong($1)?; >> and can use the bindings like: >> >> var a : PyObject = ?. >> var b: PyObject = ... >> var sum = add (a,b); > > How does it deal with reference counting and exception handling? The Felix binding generates C. So the answer is, the same way as you would in C. However the above is a low level binding, one to one with C. Higher level stuff is possible. Won?t know until the binding is operational. My script is basically working translating the *.pxd files in Cython. (Just the functions, the rest can be done by hand). Basically I can write, say, a concurrent matrix multiplication routine and map it to Python. Of course you can do that in C as well, but Felix is a very high level language so you can do stuff you wouldn?t bother attempting in low level languages like C and C++. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Tue Jan 28 01:24:40 2020 From: skaller at internode.on.net (John Skaller2) Date: Tue, 28 Jan 2020 17:24:40 +1100 Subject: [Cython] Hello In-Reply-To: <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> Message-ID: <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> Ok, quick pass on Cython reference .. pretty good. Easy to read, good coverage, well written. Beginning to understand the design. Interesting thing is the limitations are the best clue. For example: "The ??? indicates that the value -1 only indicates a possible error. In this case, Cython generates a call to PyErr_Occurred() if the exception value is returned, to make sure it really is an error.? really tells me how Cython generates code to interface C and Python. Thanks. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Tue Jan 28 04:57:52 2020 From: skaller at internode.on.net (John Skaller2) Date: Tue, 28 Jan 2020 20:57:52 +1100 Subject: [Cython] Hello In-Reply-To: <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> Message-ID: What?s a ?descr? when its at home? There are some special words in pxd files like ?list? which mean PyObject that happens to be a list. Is there a list of these somewhere? ? John Skaller skaller at internode.on.net From stefan_ml at behnel.de Tue Jan 28 05:07:33 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 28 Jan 2020 11:07:33 +0100 Subject: [Cython] Hello In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> Message-ID: John Skaller2 schrieb am 28.01.20 um 10:57: > What?s a ?descr? when its at home? A descriptor, a special protocol in Python. https://docs.python.org/3/howto/descriptor.html > There are some special words in pxd files like ?list? which mean > PyObject that happens to be a list. Is there a list of these somewhere? See Builtin.py. Stefan From skaller at internode.on.net Tue Jan 28 07:33:43 2020 From: skaller at internode.on.net (John Skaller2) Date: Tue, 28 Jan 2020 23:33:43 +1100 Subject: [Cython] Hello In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> Message-ID: > On 28 Jan 2020, at 21:07, Stefan Behnel wrote: > > John Skaller2 schrieb am 28.01.20 um 10:57: >> What?s a ?descr? when its at home? > > A descriptor, a special protocol in Python. > > https://docs.python.org/3/howto/descriptor.html Got it, thanks! > >> There are some special words in pxd files like ?list? which mean >> PyObject that happens to be a list. Is there a list of these somewhere? > > See Builtin.py. Got it, thanks! Very useful file. I?m curious about this comment in that file: # This conflicts with the C++ bool type, and unfortunately # C++ is too liberal about PyObject* <-> bool conversions, # resulting in unintuitive runtime behavior and segfaults. # ("bool", "PyBool_Type", []), I can believe it. Bool is a mess in C, twice as bad in C++ .. and exponentially screwed up in Felix :-) The problem is compatibility in bindings.. The thing is I did translate bint in pxd files to Felix bool, which is a bit, embedded in a 64 bit unsigned int .. different to both C and C++ :-) Now you have me worried! Any more detailed explanation of this issue? ? John Skaller skaller at internode.on.net From skaller at internode.on.net Tue Jan 28 14:37:41 2020 From: skaller at internode.on.net (John Skaller2) Date: Wed, 29 Jan 2020 06:37:41 +1100 Subject: [Cython] Hello In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> Message-ID: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> So I just had a thought, I might try something to enhance the bindings. Not sure if it will work. My system supports subtyping. For bindings and other nominal types you can simple define a subtyping coercion. Now in Python we have subtyping rules like: dictionary -> mapping -> object If I implement that with just static casts for coercions, then say, given a dictionary and a function accepting a mapping argument, you can pass a dictionary or mapping, but not an object. If you want to pass an object you have to explicitly coerce it. The *problem* with this idea I think is that from memory the reference counting rules for PyObject and dictionaries are not the same as for methods on abstractions. Cython must do something like this, right? And get the ref counting right. ? John Skaller skaller at internode.on.net From greg.ewing at canterbury.ac.nz Tue Jan 28 17:31:35 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 29 Jan 2020 11:31:35 +1300 Subject: [Cython] Hello In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> Message-ID: <08acacff-010e-0561-c2ba-d18e587a0227@canterbury.ac.nz> On 29/01/20 1:33 am, John Skaller2 wrote: > > # This conflicts with the C++ bool type, and unfortunately > # C++ is too liberal about PyObject* <-> bool conversions, > # resulting in unintuitive runtime behavior and segfaults. > # ("bool", "PyBool_Type", []), > > Now you have me worried! Any more > detailed explanation of this issue? I think it's just that the name "bool" refers to different things in Python and C++, and trying to unify them would lead to problems. So Cython has chosen to use "bool" to mean the Python type, and introduced "bint" for the C++ notion of a bool. This is consistent with the general philosophy that the Cython language is an extension of Python, so where there is a conflict between Python smantics and C or C++ semantics, Python wins. -- Greg From greg.ewing at canterbury.ac.nz Wed Jan 29 00:28:15 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 29 Jan 2020 18:28:15 +1300 Subject: [Cython] Hello In-Reply-To: <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> Message-ID: <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> On 29/01/20 8:37 am, John Skaller2 wrote: > If I implement that with just static casts for coercions, then say, > given a dictionary and a function accepting a mapping argument, > you can pass a dictionary or mapping, but not an object. > If you want to pass an object you have to explicitly coerce it. That sounds fine. Cython follows similar rules. > The *problem* with this idea I think is that from memory the reference counting > rules for PyObject and dictionaries are not the same as for methods > on abstractions. The vast majority of CPython API functions follow consistent refcounting rules -- they take borrowed references as arguments, and return new references as return values. However, a few of the low-level functions that operate on specific types follow different rules. Cython knows about these and treats them specially. For your system, you may need a way of annotating function declarations with refcounting rules where they differ from the defaults. -- Greg From skaller at internode.on.net Wed Jan 29 01:36:34 2020 From: skaller at internode.on.net (John Skaller2) Date: Wed, 29 Jan 2020 17:36:34 +1100 Subject: [Cython] Hello In-Reply-To: <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> Message-ID: > > For your system, you may need a way of annotating function > declarations with refcounting rules where they differ from > the defaults. Yeah, and i have to encode that to some extent in the type system. However I have two techs to play with: Felix and C++. Both can, for example, do subtyping. In addition, going pxd [Cython] -> C -> Felix allow safely and efficiently extending the ?Python C API?. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Wed Jan 29 21:47:01 2020 From: skaller at internode.on.net (John Skaller2) Date: Thu, 30 Jan 2020 13:47:01 +1100 Subject: [Cython] Status In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> Message-ID: <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> Ok, I have processed most of the Cython/Include/python/*.pxd files. Seems these were mechanically derived from docs? Because some of the bugs in the docs are also in these files. Not sure what you would like to do so I?ll ask here before doing github issues. 1. Its not all there. I couldn?t find the API for generators for example. If the API is extracted from the docs the script could be run again. 2. pyport is plain wrong. It contains conflicting C typedefs. 3. Some of the files describe extension modules (datetime, array) rather than the core API. 4. ../cython/Cython/Includes/cpython/weakref.pxd: bint PyWeakref_CheckProxy(ob) ob should be PyObject* there?s a couple more (maybe in ref as well). ==== Python keeps changing the API. Its not clear exactly which version is represented. I have the same issue, i have no idea at the moment how to deal with it. ===== I?m thinking to do some test runs of Cython on some Python code. I?ll be running Cython as a translator directly from the command line. Now when I built Cython before with ?make? I saw lots of ?python2.7? in the output which is not what I want to process. My Felix test code is executing plain C code bound againt a brew installed Python run time library, the meta data required to link that is this: Name: Pycore Description: Macosx bindings to Python shared run time library core. cflags: -I/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/include/python3.6m includes: '"Python.h"' provides_dlib: -L/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib -lpython3.6m provides_slib: -L/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib -lpython3.6m And I can run the corresponding Python: ~/felix>/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6m Python 3.6.5 (default, Apr 25 2018, 14:26:36) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> And here is a test program written in Felix that uses the bindings: println$ Py_GetVersion(); Py_Initialize(); var x : PyObject = PyLong_FromLong (42); var y : long = PyLong_AsLong (x); println$ PyLong_Check x; println$ y; println$ (f"%x" header_version); and here?s the output: ~/felix>flx pycheck.flx 3.6.5 (default, Apr 25 2018, 14:26:36) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] true 42 30605f0 Note this code is generating a shared library BUT it is being run by Felix, it is NOT being loaded as a module under Python. I will check that works next. So I?m showing all this because I would now like to run Cython on a little test script using the same run time library as above, and perhaps the same Python interpreter. I have too many Python?s installed all over the place to really know what?s what. So I would appreciate step by step instructions on how to build and use Cython so it uses and generates code binding to the exact same Python/run time as above. I guess I?ll test some kind of hello world program. === I?m expecting that on OSX and Windows, CPython extensions will work, but on Linux they will fail. This is because last time I looked, extensions were built assuming a flat linker namespace with the API provided by the Python executable. Indeed, I?m not sure if you can even get a Python executable on Linux that isn?t statically linked to the run time. However the shared library extensions I generate will always be linked against a shared library. My build tech does not permit flat namespaces for dynamic linkage. I know you CAN get Python run time as a shared library on Debian based systems, but I don?t think you can get a stub Python executable. Its essential that all the global data is held by the library not the executable, otherwise it won?t be shared by libraries that link to it. However stuff may have changed recently. I can of course build Python from source myself and hack it to build correctly, its not that hard, but Felix clients will not be able to do that. This issue will impact Cython uses exactly the same way for the same reason. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Thu Jan 30 01:20:27 2020 From: skaller at internode.on.net (John Skaller2) Date: Thu, 30 Jan 2020 17:20:27 +1100 Subject: [Cython] Status In-Reply-To: <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> Message-ID: <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> Just FYI, trying to load my extension I get ?ModuleNotFound?. A test *.py file in the same directory can be imported. A check on the shared library (dylib on MacOS) shows that the required function is exported. Maybe my layout of the module table is wrong, its hard to say given so much information (sarcasm). The Python function is _fI64918_ptest. The function _fI64918_ptest executes correctly when called from C. If it crashed when called from Python, I wouldn?t be surprised, but the problem is more basic, the extension can?t be loaded. The generated code below compiles fine. The FLX_EXPORT there is the visibility control** static PyMethodDef pycheck_methods [] = { {"ptest", _fI64918_ptest, METH_VARARGS, "/Users/skaller/felix/pycheck.flx: line 98, cols 1 to 61"}, {NULL, NULL, 0, NULL} }; static PyModuleDef pycheck_module = { PyModuleDef_HEAD_INIT, // m_base "pycheck", // m_name "pycheck generated by Felix ", // m_doc -1, // m_size pycheck_methods, // m_methods 0, // m_reload 0, // m_traverse 0, // m_clear 0 // m_free }; extern "C" FLX_EXPORT PyObject *PyInit_pycheck() { return PyModule_Create(&pycheck_module);} ** Python gets the visibilty control wrong for Linux. That?s because Python is built incorrectly on Linux. But I?m running MacOS so it should work, shouldn?t it? ? John Skaller skaller at internode.on.net From skaller at internode.on.net Thu Jan 30 02:44:02 2020 From: skaller at internode.on.net (John Skaller2) Date: Thu, 30 Jan 2020 18:44:02 +1100 Subject: [Cython] Status In-Reply-To: <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> Message-ID: > On 30 Jan 2020, at 17:20, John Skaller2 wrote: > > Just FYI, trying to load my extension I get ?ModuleNotFound?. using Cython I get a different result. ~/felix>cat oldtest.py def testit(): print("Testit?) Note the original name was test.py, changed so it cannot be loaded. Under the original name: ~/felix>python ../cython/cython.py test.py /Users/skaller/cython/Cython/Compiler/Main.py:344: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /Users/skaller/felix/test.py tree = Parsing.p_module(s, pxd, full_module_name) That made a C file. Now to compile it: ~/felix>clang -I /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/include/python3.6m/ -L /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib -lpython3.6m -dynamic -shared -o test.dylib test.c ~/felix>ls test.dylib test.dylib ~/felix>mv test.py oldtest.py ~/felix>/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/bin/python3.6m Python 3.6.5 (default, Apr 25 2018, 14:26:36) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import test Traceback (most recent call last): File "", line 1, in ImportError: bad magic number in 'test': b'\x03\xf3\r\n? Well, this is a better result than from the C generated by Felix. At least the extension module actually got found, even if it failed a consistency check. Any idea what?s happening? [the Cython seems to have been built for 2.7, I don?t know if that makes any difference: it?s run by command ?python? which is Python 2.7. But the languagfe of the translator isn?t necessarily related to the target of its output .. there?s no reference for the command line compiler] ? John Skaller skaller at internode.on.net From yury at shurup.com Thu Jan 30 02:00:34 2020 From: yury at shurup.com (Yury V. Zaytsev) Date: Thu, 30 Jan 2020 08:00:34 +0100 Subject: [Cython] Status In-Reply-To: <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> References: <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> Message-ID: <265CAF41-3657-494C-9DC0-522F47F495EE@shurup.com> On 30. Jan 2020, at 03:52, John Skaller2 wrote: > > However the shared library extensions I generate will always be linked > against a shared library. My build tech does not permit flat namespaces > for dynamic linkage. I know you CAN get Python run time as a shared > library on Debian based systems, but I don?t think you can get > a stub Python executable. Its essential that all the global data is held > by the library not the executable, otherwise it won?t be shared by > libraries that link to it. Users of MPI libraries have some of the same problems - maybe you could have a look at the mpi4py - coincidentally it?s also a heavy user of Cython, so a lot of inspiration can be found in the sources. https://mpi4py.readthedocs.io/en/stable/appendix.html#mpi-enabled-python-interpreter Sent from my iPad -------------- next part -------------- An HTML attachment was scrubbed... URL: From skaller at internode.on.net Thu Jan 30 02:58:34 2020 From: skaller at internode.on.net (John Skaller2) Date: Thu, 30 Jan 2020 18:58:34 +1100 Subject: [Cython] Status In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> Message-ID: <9ED292F5-B12E-45C2-B8E5-FAD14C6DBDD0@internode.on.net> > On 30 Jan 2020, at 18:44, John Skaller2 wrote: > > > >> On 30 Jan 2020, at 17:20, John Skaller2 wrote: >> >> Just FYI, trying to load my extension I get ?ModuleNotFound?. > > ~/felix>/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/bin/python3.6m > Python 3.6.5 (default, Apr 25 2018, 14:26:36) > [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin > Type "help", "copyright", "credits" or "license" for more information. >>>> import test > Traceback (most recent call last): > File "", line 1, in > ImportError: bad magic number in 'test': b'\x03\xf3\r\n? Oh, sorry! This was a *pyc file hanging around. After deleting it Python happily loaded the wrong module because I picked a bad name. So .. /Users/skaller/cython/Cython/Compiler/Main.py:344: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /Users/skaller/felix/oldtest.py tree = Parsing.p_module(s, pxd, full_module_name) ~/felix>clang -I /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/include/python3.6m/ -L /usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/lib -lpython3.6m -dynamic -shared -o oldtest.dylib oldtest.c ~/felix>mv oldtest.py oldertest.py ~/felix>/usr/local/opt/python/Frameworks/Python.framework/Versions/3.6/bin/python3.6m Python 3.6.5 (default, Apr 25 2018, 14:26:36) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import oldtest Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'oldtest' >>> which is the same problem my Felix generated C is giving. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Thu Jan 30 03:15:22 2020 From: skaller at internode.on.net (John Skaller2) Date: Thu, 30 Jan 2020 19:15:22 +1100 Subject: [Cython] Status In-Reply-To: <265CAF41-3657-494C-9DC0-522F47F495EE@shurup.com> References: <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <265CAF41-3657-494C-9DC0-522F47F495EE@shurup.com> Message-ID: <4B6150EB-0782-4707-B96A-ACF04C16686C@internode.on.net> > On 30 Jan 2020, at 18:00, Yury V. Zaytsev wrote: > > On 30. Jan 2020, at 03:52, John Skaller2 wrote: >> >> However the shared library extensions I generate will always be linked >> against a shared library. My build tech does not permit flat namespaces >> for dynamic linkage. I know you CAN get Python run time as a shared >> library on Debian based systems, but I don?t think you can get >> a stub Python executable. Its essential that all the global data is held >> by the library not the executable, otherwise it won?t be shared by >> libraries that link to it. > > Users of MPI libraries have some of the same problems - maybe you could have a look at the mpi4py - coincidentally it?s also a heavy user of Cython, so a lot of inspiration can be found in the sources. > > https://mpi4py.readthedocs.io/en/stable/appendix.html#mpi-enabled-python-interpreter Thanks. That ?rebuild Python from source? is also my main worry. Luckily, I previously found, as long as you have the runtime, its easy to build a stub like the one shown in the above page. The problem is prebuilt binary extensions don?t work, unless the symbol table of the library is imported into the executable so they can be re-exported into the extensions. Which requires a flat namespace at least for the executable. With my system, the executable is pre-built but it has no external dependencies except C library with dlopen (or Windows equivalent). Even if that used a flat namespace it wouldn?t help, because the shared libs I generate always link to their dependencies with a two level namespace, so the dependence on Python run time cannot re-export the run time library symbols. That, after all, is the whole point of using a two level namespace. Of course, I can also do static links which is intrinsically using a flat namespace. So I could change it to use a flat namespace. But sometimes progress requires avoiding stepping back decades into the medieval world :-) [No relation to Vasily Grigoryevich Zaytsev? Just curious ..] ? John Skaller skaller at internode.on.net From skaller at internode.on.net Thu Jan 30 04:31:05 2020 From: skaller at internode.on.net (John Skaller2) Date: Thu, 30 Jan 2020 20:31:05 +1100 Subject: [Cython] Status In-Reply-To: <4B6150EB-0782-4707-B96A-ACF04C16686C@internode.on.net> References: <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <265CAF41-3657-494C-9DC0-522F47F495EE@shurup.com> <4B6150EB-0782-4707-B96A-ACF04C16686C@internode.on.net> Message-ID: OMG. I don?t believe it. Renaming the binary from *.dylib to *.so fixed it. Both the Cython and Felix shared libraries can be imported and both run. Sorry for the noise. Just didn?t occur to me that Python would want *.so filenames on MacOS. Until I remembered seeing some in Cython build directory .. Here is the Felix program, just for interest, excluding the binding code, note I defined ?PyObject? in Felix to be ?PyObject*? in C. /////// fun ptest(q:PyObject,w:PyObject):PyObject { println$ Py_GetVersion(); Py_Initialize(); var x : PyObject = PyLong_FromLong (42); var y : long = PyLong_AsLong (x); println$ PyLong_Check x; println$ y; println$ (f"%x" header_version); return x; } C_hack::ignore (ptest(Py_None,Py_None)); export python fun ptest of (PyObject * PyObject) as "ptest?; ////////// The ptest fun at the start is an ordinary Felix function that just happens to use the CPython API bindings I just generated to make a function with the type actually required for CPython. The C_hack::ignore (ptest (Py_None, Py_None)); generates a C function call that actually runs it from C. The hack is needed because Felix function are not allowed to have side effects, and, if the result of a function is unused, the function then cannot serve any useful purpose, and is deleted. The ?hack? fools the compiler into believing the result of the function is actually used. That call is done as part of the library initialisation. The final line is the interesting one. It generates a wrapper for the function ptest, which is a plain C function AND it tells the compiler, there is a Python function floating about that has to be put into a Python Module table. So the compiler builds a module table, and puts the function in it, and then defines PyInit_modulename() which returns the module table address, and exports it. When imported from Python, the initialisation is not done, but the function can be run from the module dictionary., So this shared library can be used BOTH from C, and from Python. It?s a Felix program AND a Python extenion as well. The idea has been, you can write ultra-high performance code in Felix, and then with some work to make a CPython binding to the operations, you can make the resulting binary library accessible not only to C (and C++), but also to Python, by simply tell the compiler to export functions to python. ? John Skaller skaller at internode.on.net From greg.ewing at canterbury.ac.nz Thu Jan 30 06:23:40 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 31 Jan 2020 00:23:40 +1300 Subject: [Cython] Status In-Reply-To: <9ED292F5-B12E-45C2-B8E5-FAD14C6DBDD0@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> <9ED292F5-B12E-45C2-B8E5-FAD14C6DBDD0@internode.on.net> Message-ID: <3a1fbce6-3a36-6a0e-d7e8-98074ee66fdb@canterbury.ac.nz> On 30/01/20 8:58 pm, John Skaller2 wrote: >>>> import oldtest > Traceback (most recent call last): > File "", line 1, in > ModuleNotFoundError: No module named 'oldtest' What happens if you use a .so extension instead of .dylib? -- Greg From stefan_ml at behnel.de Thu Jan 30 08:22:56 2020 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 30 Jan 2020 14:22:56 +0100 Subject: [Cython] Status In-Reply-To: <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> Message-ID: John Skaller2 schrieb am 30.01.20 um 03:47: > Ok, I have processed most of the Cython/Include/python/*.pxd files. > > Seems these were mechanically derived from docs? They were manually copied over time. > Because some of the bugs in the docs are also in these files. Not sure what you > would like to do so I?ll ask here before doing github issues. > > 1. Its not all there. I couldn?t find the API for generators for example. > If the API is extracted from the docs the script could be run again. > > 2. pyport is plain wrong. It contains conflicting C typedefs. PRs welcome. > 3. Some of the files describe extension modules (datetime, array) > rather than the core API. That's fine. > 4. ../cython/Cython/Includes/cpython/weakref.pxd: bint PyWeakref_CheckProxy(ob) > > ob should be PyObject* No, the declaration looks correct to me. The input is an object. It does not seem helpful to me to require users to do a cast. > there?s a couple more (maybe in ref as well). PRs welcome. Stefan From skaller at internode.on.net Thu Jan 30 15:47:31 2020 From: skaller at internode.on.net (John Skaller2) Date: Fri, 31 Jan 2020 07:47:31 +1100 Subject: [Cython] Status In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> Message-ID: <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> > On 31 Jan 2020, at 00:22, Stefan Behnel wrote: > >> Seems these were mechanically derived from docs? > > They were manually copied over time. Ouch. A lot of work. >> 2. pyport is plain wrong. It contains conflicting C typedefs. > > PRs welcome. Is this your prefered method (pull request)? >> 4. ../cython/Cython/Includes/cpython/weakref.pxd: bint PyWeakref_CheckProxy(ob) >> >> ob should be PyObject* > > No, the declaration looks correct to me. The input is an object. It does > not seem helpful to me to require users to do a cast. I don?t understand. ob isn?t a type, is it? A type is required. Else where ?object? is used as an alias for PyObject*. It also suggests standard ref counting is required whereas plain PyObject* suggests non-standard ref counting. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Thu Jan 30 16:00:08 2020 From: skaller at internode.on.net (John Skaller2) Date: Fri, 31 Jan 2020 08:00:08 +1100 Subject: [Cython] Status In-Reply-To: <3a1fbce6-3a36-6a0e-d7e8-98074ee66fdb@canterbury.ac.nz> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> <9ED292F5-B12E-45C2-B8E5-FAD14C6DBDD0@internode.on.net> <3a1fbce6-3a36-6a0e-d7e8-98074ee66fdb@canterbury.ac.nz> Message-ID: <31F56C6D-FC99-48FF-9CD1-160B5E52267A@internode.on.net> > On 30 Jan 2020, at 22:23, Greg Ewing wrote: > > On 30/01/20 8:58 pm, John Skaller2 wrote: > >>>> import oldtest > > Traceback (most recent call last): > > File "", line 1, in > > ModuleNotFoundError: No module named 'oldtest' > What happens if you use a .so extension instead of .dylib? It works. Why Python would look for a linux extension only on MacOS I have no idea. A symlink appears to work too. Was tearing hare out for many hours. ? John Skaller skaller at internode.on.net From greg.ewing at canterbury.ac.nz Fri Jan 31 00:43:02 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 31 Jan 2020 18:43:02 +1300 Subject: [Cython] Status In-Reply-To: <31F56C6D-FC99-48FF-9CD1-160B5E52267A@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> <9ED292F5-B12E-45C2-B8E5-FAD14C6DBDD0@internode.on.net> <3a1fbce6-3a36-6a0e-d7e8-98074ee66fdb@canterbury.ac.nz> <31F56C6D-FC99-48FF-9CD1-160B5E52267A@internode.on.net> Message-ID: <830ebd9e-c884-af1f-5a5a-6e3945f85f79@canterbury.ac.nz> On 31/01/20 10:00 am, John Skaller2 wrote: > It works. Why Python would look for a linux extension only on MacOS I have no idea. It's not really a Linux extension. It goes back to when shared libraries first appeared in BSD, from which large parts of MacOSX are derived, and MacOSX also recognises it as a valid extension for dynamically linked object files. I guess the Python developers didn't think it was worth making a distinction between different flavours of unix in this area. The mystery to me is why MacOSX introduced .dylib instead of sticking with .so. -- Greg From greg.ewing at canterbury.ac.nz Fri Jan 31 00:51:04 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 31 Jan 2020 18:51:04 +1300 Subject: [Cython] Status In-Reply-To: <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> Message-ID: On 31/01/20 9:47 am, John Skaller2 wrote: >>> 2. pyport is plain wrong. It contains conflicting C typedefs. >> >> PRs welcome. > > Is this your prefered method (pull request)? I'm sure PRs are very welcome, but at the least you could give us some idea of what these conflicting typedefs are! >>> ob should be PyObject* >> >> No, the declaration looks correct to me. The input is an object. > > I don?t understand. ob isn?t a type, is it? A type is required. It's a (dummy) parameter name. Cython defaults to "object" when a type isn't specified. Looking at the other declarations in that file, it was probably *meant* to say "object ob", but it's not wrong -- it still works that way. > Else where ?object? is used as an alias for PyObject*. > It also suggests standard ref counting is required whereas > plain PyObject* suggests non-standard ref counting. And you want standard refcounting here. -- Greg From skaller at internode.on.net Fri Jan 31 06:10:21 2020 From: skaller at internode.on.net (John Skaller2) Date: Fri, 31 Jan 2020 22:10:21 +1100 Subject: [Cython] Status In-Reply-To: <830ebd9e-c884-af1f-5a5a-6e3945f85f79@canterbury.ac.nz> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <27B7B984-BBE4-48DB-8805-2C3574E11DA7@internode.on.net> <9ED292F5-B12E-45C2-B8E5-FAD14C6DBDD0@internode.on.net> <3a1fbce6-3a36-6a0e-d7e8-98074ee66fdb@canterbury.ac.nz> <31F56C6D-FC99-48FF-9CD1-160B5E52267A@internode.on.net> <830ebd9e-c884-af1f-5a5a-6e3945f85f79@canterbury.ac.nz> Message-ID: <8D0D9FF0-6391-4CF2-84BF-9C89417E0D66@internode.on.net> > The mystery to me is why MacOSX introduced .dylib instead of > sticking with .so. There were *.so files and hacks to load them. But the structure od dylib is different and uses a slightly different loader, dyld. I guess they wanted to make a distinction. The had some kind of Obj C dynamic plugin things as well. Software history is full of regrets. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Fri Jan 31 06:25:01 2020 From: skaller at internode.on.net (John Skaller2) Date: Fri, 31 Jan 2020 22:25:01 +1100 Subject: [Cython] Status In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> Message-ID: <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> > On 31 Jan 2020, at 16:51, Greg Ewing wrote: > > On 31/01/20 9:47 am, John Skaller2 wrote: > >>>> 2. pyport is plain wrong. It contains conflicting C typedefs. >>> >>> PRs welcome. >> Is this your prefered method (pull request)? > > I'm sure PRs are very welcome, but at the least you could > give us some idea of what these conflicting typedefs are! The file is small: cdef extern from "Python.h": ctypedef int int32_t ctypedef int int64_t ctypedef unsigned int uint32_t ctypedef unsigned int uint64_t Obviously this is an incorrect translation of the original source. One of each pair may well be correct. But its impossible both are. Defining a symbol defined in the C99 standard seems like a bad idea. Python?s pyport.h actually says: #include .. #define PY_UINT32_T uint32_t #define PY_UINT64_T uint64_t /* Signed variants of the above */ #define PY_INT32_T int32_t #define PY_INT64_T int64_t ? ? John Skaller skaller at internode.on.net From frank.schlimbach at intel.com Fri Jan 31 06:29:36 2020 From: frank.schlimbach at intel.com (Schlimbach, Frank) Date: Fri, 31 Jan 2020 11:29:36 +0000 Subject: [Cython] Adding GPU support to cython In-Reply-To: References: Message-ID: Hi, I opened a feature ticket: https://github.com/cython/cython/issues/3342 It describes my current prototype based on OpenMP. Any feedback? Also, I would like to do some more advanced analysis to improve the map-clauses. I do not want to go to a complex index analysis or alike, but a simple access analysis should cover many cases. All I would like to figure out is if a given variable (memview) was used (other than instantiated) before and/or after the device/parallel/device) block and ideally of a use was definitely a read-only. Any suggestion/hint how to do that? Thanks frank -----Original Message----- From: cython-devel On Behalf Of Schlimbach, Frank Sent: Friday, January 24, 2020 12:55 PM To: Core developer mailing list of the Cython compiler Subject: Re: [Cython] Adding GPU support to cython Hi Stefan, thanks for your response. Good to hear this is still of interest. Yes, I realized these are rather old CEPs. I spent some time with looking into the Cython code and concluded that it'd be the most consistent (and simplest) approach to stick with OpenMP and use it's offload pragmas (e.g. 'target' introduced in 4.5). Given a properly setup compiler this would in theory only require one or two compiler flags to enable offloading. I even have a first prototype which generates code that existing compilers seem to swallow. It's not ready for a PR since I have not been able to get it linked an run on GPU and I wanted to get some general feedback first. You can find the code on my offload branch https://github.com/fschlimb/cython/tree/offload (it's wip so please apologize that not all comments have been updated yet to reflect my changes). Here's what it does: - accept a new 'with' directive 'device' which marks a region/block to be offloaded to a device (OpenMP target) - I also considered extending 'gil' or 'parallel' to accept an optional 'device' argument but an extra directive seemed more general/flexible to also allow non-parallel code - I don't believe we should try to automate offloading right now. Once we have something that works on explicit demand we can still think about a performance model and auto-enable offloading. - the DeviceWithBlockNode is added to the 'parallel stack' and can occur only as the outmost parallel directive - a 'with device()' requires 'nogil' - a 'with device()' will create a new scope annotated with a '#pragma omp target' - all variables which get assigned within the 'with device()' block are currently mapped as 'tofrom' - all other variables used are mapped as 'to' - identifying 'from' candidates is harder and not yet done (need to know that there is required allocation but no assignment before the 'with device()' block) - identifying 'alloc' candidates would also need additional analysis (e.g. not used outside the 'device()' block) - all object mode stuff (like exceptions for error handling) are currently disabled in a 'with device()' block Example: def f(int[:,::1] X): cdef int v = 1 cdef int i with gil, device(), parallel(): for i in prange(4): X[i] = v the 'with device' block becomes something like (simplified) { size_t __pyx_v_X__count = __pyx_v_X.shape[0]*__pyx_v_X.shape[1]; #pragma omp target map(to: __pyx_v_v) map(tofrom: __pyx_v_i , __pyx_v_X.data[0:__pyx_v_X__count], __pyx_v_X.memview, __pyx_v_X.shape, __pyx_v_X.strides, __pyx_v_X.suboffsets) { #pragma omp parallel #pragma omp for firstprivate(__pyx_v_i) lastprivate(__pyx_v_i) for((__pyx_v_i=0; __pyx_v_i<4; ++__pyx_v_i) { __pyx_v_X[__pyx_v_i] = __pyx_v_v; } } } There are lots of things to be added and improved, in particular I am currently adding an optional argument 'map' to 'device()' which allows manually setting the map-clauses for each variable. This is necessary to allow not only optimizations but also sending only partial array data to/from the device (like when the device memory cannot hold an entire array the developer would block the computation). We can probably add some magic for simple cases but there is probably no solution for the general problem of determining the accessed index-space. Among others, things to also look at include - non-contiguous arrays/memviews - overlapping arrays/memviews - keeping data on the device between 'with device()' blocks (USM (unified shared memory) or omp target data?) - error handling - tests - docu/comments I found that the functionality I needed to touch is somewhat scattered around the compiler pipeline. It might be worth thinking about restructuring a few things to make the whole OpenMP/parallel/offload stuff more maintainable. Of course you might see other solutions than mine which make this simpler. Any thoughts/feedback/usecases appreciated frank -----Original Message----- From: cython-devel On Behalf Of Stefan Behnel Sent: Friday, January 24, 2020 11:22 AM To: cython-devel at python.org Subject: Re: [Cython] Adding GPU support to cython Hi Frank, sorry for leaving this unresponded for a while. I'm far from an expert in this, but it looks like no-one else jumped in, so here's my response. Schlimbach, Frank schrieb am 06.01.20 um 12:09: > I would like to work on extending cython with a way to offload cython code to a GPU. I found to related CEPs (https://github.com/cython/cython/wiki/enhancements-opencl and https://github.com/cython/cython/wiki/enchancements-metadefintions). So, just for a bit of context, these CEPs were written a decade ago, and Cython's code base, feature set, and the general Python ecosystem have evolved a lot since then. For example, "cython.parallel" and "prange()" weren't even implemented back then. > My current thinking is that a solution along the OpenCL CEP is most effective, it does require many code changes and seems to provide a good tradeoff between usability and efficiency. > > I would like to suggest a few modifications to this approach, like > > * using SYCL instead of OpenCL to closely follow existing parallel/prange semantics more easily > * selecting the device (CPU, GPU) per region rather than per file > * maybe allowing calling appropriately annotated and written external functions > > I would be very grateful for any thoughts about this topic in general and for any advice on how to approach this so that a solution is found that is most broadly useful and most cythonic. It would definitely be cool to generate GPU support from the existing Cython patterns, in addition to the OpenMP code that we already generate. If that can be done, then users could enable GPU support by adding a C compiler define to their CFLAGS (rather than rerunning Cython), or even select between the two versions at runtime. If the GPU support is per region, then how is the code section shipped to the GPU? Is the infrastructure for this provided by the OpenCL framework or does the user or the module need to set something up in addition? Finally, generally speaking: - PR welcome - simple approach preferred (at least to get this started and prototyped) - discussion welcome on this mailing list - GitHub feature ticket seems to be missing, with a link to the ML thread https://mail.python.org/pipermail/cython-devel/2020-January/005262.html Stefan _______________________________________________ cython-devel mailing list cython-devel at python.org https://mail.python.org/mailman/listinfo/cython-devel Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 _______________________________________________ cython-devel mailing list cython-devel at python.org https://mail.python.org/mailman/listinfo/cython-devel Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 From skaller at internode.on.net Fri Jan 31 06:34:06 2020 From: skaller at internode.on.net (John Skaller2) Date: Fri, 31 Jan 2020 22:34:06 +1100 Subject: [Cython] Status In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> Message-ID: <87EA24CB-73B8-4C0B-95DE-85BB02690ECB@internode.on.net> >>>> ob should be PyObject* >>> >>> No, the declaration looks correct to me. The input is an object. >> I don?t understand. ob isn?t a type, is it? A type is required. > > It's a (dummy) parameter name. Cython defaults to "object" when a > type isn't specified. > > Looking at the other declarations in that file, it was probably > *meant* to say "object ob", but it's not wrong -- it still works > that way. Ok, but now the syntax is made very context sensitive. To interpret it correctly, you have to know ?ob? is not a type. And the Python docs make exactly the same mistake. In C this would not work because there is no default type, so the Python docs are wrong because they?re supposedly documenting C. [The only case it could be correct would be if the symbol were a macro] And my translator script got fooled, because it assumes any single identifier used as a parameter is a type, and if two words are used, the first is a type and the second can be discarded, except in the special case ?unsigned int?. Note, I?m just trying to help by bringing up inconsistencies, which are things my simplistic translator script can?t handle. ? John Skaller skaller at internode.on.net From greg.ewing at canterbury.ac.nz Fri Jan 31 08:36:05 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Feb 2020 02:36:05 +1300 Subject: [Cython] Status In-Reply-To: <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> Message-ID: <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> On 1/02/20 12:25 am, John Skaller2 wrote: > cdef extern from "Python.h": > ctypedef int int32_t > ctypedef int int64_t > ctypedef unsigned int uint32_t > ctypedef unsigned int uint64_t These work because Cython doesn't need to know the exact sizes of these types. All it needs to know is that they're some kind of integer so that its type checks will pass. The typedef names end up in the generated C code, and the C compiler figures out their actual sizes. > Obviously this is an incorrect translation of the original source. Extern declarations in Cython are not meant to be exact translations. They only need to tell Cython enough about the thing being declared so that it can cope. -- Greg From greg.ewing at canterbury.ac.nz Fri Jan 31 08:50:39 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Feb 2020 02:50:39 +1300 Subject: [Cython] Status In-Reply-To: <87EA24CB-73B8-4C0B-95DE-85BB02690ECB@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <87EA24CB-73B8-4C0B-95DE-85BB02690ECB@internode.on.net> Message-ID: <5f8530af-69ad-384b-7048-51764143cb7e@canterbury.ac.nz> On 1/02/20 12:34 am, John Skaller2 wrote: > Ok, but now the syntax is made very context sensitive. > To interpret it correctly, you have to know ?ob? is not a type. Yes, Cython mostly follows C declaration syntax, and C also has this property. > In C this would not work because there is no default type, Yes, there is -- the default type in C is int. This is a valid function definition in C: f(x) { } It's equivalent to int f(int x) { } > And my translator script got fooled, because it assumes > any single identifier used as a parameter is a type, Then it's making an incorrect assumption. -- Greg From skaller at internode.on.net Fri Jan 31 09:03:35 2020 From: skaller at internode.on.net (John Skaller2) Date: Sat, 1 Feb 2020 01:03:35 +1100 Subject: [Cython] Status In-Reply-To: <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> Message-ID: > On 1 Feb 2020, at 00:36, Greg Ewing wrote: > > On 1/02/20 12:25 am, John Skaller2 wrote: >> cdef extern from "Python.h": >> ctypedef int int32_t >> ctypedef int int64_t >> ctypedef unsigned int uint32_t >> ctypedef unsigned int uint64_t > > These work because Cython doesn't need to know the exact > sizes of these types. All it needs to know is that they're > some kind of integer so that its type checks will pass. > The typedef names end up in the generated C code, and the > C compiler figures out their actual sizes. Ah. I see. That makes sense. So this is some kind of hack way of getting something a bit like Haskell type classes, you?re basically saying int32_t and int64_t are of class ?Integer?. This also explains the conflict for me, because Felix is the opposite: it aims to make the types of things more precise (and has actual type classes for generalisation). ? John Skaller skaller at internode.on.net From skaller at internode.on.net Fri Jan 31 09:08:32 2020 From: skaller at internode.on.net (John Skaller2) Date: Sat, 1 Feb 2020 01:08:32 +1100 Subject: [Cython] Status In-Reply-To: <5f8530af-69ad-384b-7048-51764143cb7e@canterbury.ac.nz> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <87EA24CB-73B8-4C0B-95DE-85BB02690ECB@internode.on.net> <5f8530af-69ad-384b-7048-51764143cb7e@canterbury.ac.nz> Message-ID: <1BAF30A1-36C5-460E-B643-21A3C9B95A93@internode.on.net> > > Yes, there is -- the default type in C is int. I don?t think that is true in C99 but I?m not sure. Its definitely not allowed in C++. I know because I actually moved the motion on the C++ ISO committee to disallow it :-) In any case its a bad idea in an interface specification even if it?s legal. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Fri Jan 31 09:17:24 2020 From: skaller at internode.on.net (John Skaller2) Date: Sat, 1 Feb 2020 01:17:24 +1100 Subject: [Cython] Size of output In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> Message-ID: <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> When I ran Cython on a two line Python function I got this from wc: 4276 13798 161338 oldtest.c It took a while to actually find the implementation of the function. A lot of the emitted code appeared to be run time and compile time support code which wasn?t actually used. Eliminating stuff that isn?t required with dependency tracking is nontrivial, and not much use whereas a single self contained compilable C files is very useful. Is there an option to use an #include for the standard stuff? ? John Skaller skaller at internode.on.net From greg.ewing at canterbury.ac.nz Fri Jan 31 17:49:16 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Feb 2020 11:49:16 +1300 Subject: [Cython] Status In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> Message-ID: On 1/02/20 3:03 am, John Skaller2 wrote: > > So this is some kind of hack way > of getting something a bit like Haskell type classes, > you?re basically saying int32_t and int64_t are of class ?Integer?. I suppose you could think of it that way, but it's really not that formal. > This also explains the conflict for me, because Felix is the opposite: > it aims to make the types of things more precise (and has actual > type classes for generalisation). To define them any more precisely, Cython would need to know how things vary depending on the platform, which would mean conditional compilation, etc. It's much easier to leave all that up to the C compiler and system headers. It also ensures that there can't be any mismatch between the two. -- Greg From greg.ewing at canterbury.ac.nz Fri Jan 31 18:00:58 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Feb 2020 12:00:58 +1300 Subject: [Cython] Status In-Reply-To: <1BAF30A1-36C5-460E-B643-21A3C9B95A93@internode.on.net> References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <87EA24CB-73B8-4C0B-95DE-85BB02690ECB@internode.on.net> <5f8530af-69ad-384b-7048-51764143cb7e@canterbury.ac.nz> <1BAF30A1-36C5-460E-B643-21A3C9B95A93@internode.on.net> Message-ID: On 1/02/20 3:08 am, John Skaller2 wrote: > I don?t think that is true in C99 but I?m not sure. You may be right, I haven't been keeping up with all the twists and turns of recent C standards. The gcc I just tried it on allowed it, but warned about it. > In any case its a bad idea in an interface specification even if it?s legal. Perhaos in C, but I think it makes sense for types in Cython to default to object, because it deals with objects so much. It means that functions that take and return objects exclusively look like Python. -- Greg From greg.ewing at canterbury.ac.nz Fri Jan 31 18:17:29 2020 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Feb 2020 12:17:29 +1300 Subject: [Cython] Size of output In-Reply-To: <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> References: <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> Message-ID: On 1/02/20 3:17 am, John Skaller2 wrote: > When I ran Cython on a two line Python function I got this from wc: > > 4276 13798 161338 oldtest.c That seems a bit excessive. > A lot of the emitted code appeared to be run time and compile > time support code which wasn?t actually used. Not sure what's going on there. Pyrex made efforts to only include support code that was actually used, but Cython has changed a lot since then and I haven't been following its development closely. Either it's slipped on that, or the support code has become more bloated. Can you remove any of it and still have it compile? If so, filing a bug report might be useful. > Is there an option to use an #include for the standard stuff? There are upsides and downsides to that as well. The way things are, the generated file is self-contained, and can be shipped without worrying about it becoming disconnected from a compatible version of the include file. This is important when details of the support code can change without notice between Cython releases. -- Greg From robertwb at gmail.com Fri Jan 31 18:49:09 2020 From: robertwb at gmail.com (Robert Bradshaw) Date: Fri, 31 Jan 2020 15:49:09 -0800 Subject: [Cython] Size of output In-Reply-To: References: <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> Message-ID: On Fri, Jan 31, 2020 at 3:17 PM Greg Ewing wrote: > > On 1/02/20 3:17 am, John Skaller2 wrote: > > When I ran Cython on a two line Python function I got this from wc: > > > > 4276 13798 161338 oldtest.c > > That seems a bit excessive. > > > A lot of the emitted code appeared to be run time and compile > > time support code which wasn?t actually used. > > Not sure what's going on there. Pyrex made efforts to only > include support code that was actually used, but Cython > has changed a lot since then and I haven't been following > its development closely. Either it's slipped on that, or > the support code has become more bloated. Cython attempts to do the same. Taking a quick glance at an auto-generated file for an empty .pyx, we have ~200 lines of macros normalizing various C compiler issues ~500 lines defining macros to normalize across Python 2.7-3.9 ~200 lines of providing defaults for various CYTHON_ macros ~300 lines of macros for optional optimizations for CPython details (vs. using more public/pypy compatible, ... APIs) ~300 lines module setup code. Even for trivial modules, we still declare and call functions for creating globals, preparing types, etc. even if we don't have any globals, types, etc. ~300 lines exception handling and traceback creation ~700 lines conversion for basic int and string types (which we assume to be available in various utilities). Much of this is macro-heavy code, to allow maximum flexibility at C compile time, but much would get elided by the preprocessor for any particular environment. Extra utility code is inserted on an as-needed bases, e.g. function creation, various dataytype optimizations, other type conversions, etc. These are re-used within a module. A two line function could add a lot (e.g. just defining a function and its wrapper is a good chunk of code, and whatever the function does of course). I agree there's some fat that could be trimmed there, but not sure it'd be worth the effort. > Can you remove any of it and still have it compile? If > so, filing a bug report might be useful. +1 > > Is there an option to use an #include for the standard stuff? > > There are upsides and downsides to that as well. The way > things are, the generated file is self-contained, and can > be shipped without worrying about it becoming disconnected > from a compatible version of the include file. This is > important when details of the support code can change > without notice between Cython releases. +1 We have an option "common_utility_include_dir" that would create a shared utility folder into which the compiler could create (versioned) #includable files to possibly be shared across many modules, but it was never completely finished (and in particular was difficult to reconcile with cycache, which is like ccache for Cython, due to the outside references a cython artifact could then produce). We've thought of going even further and providing a shared runtime library, but that has some of the same issues (plus more, though in some cases we use the pattern where every module declares type X, but before creating its own looks to see if one was already loaded to let modules share the same internal type at runtime). From skaller at internode.on.net Fri Jan 31 21:29:05 2020 From: skaller at internode.on.net (John Skaller2) Date: Sat, 1 Feb 2020 13:29:05 +1100 Subject: [Cython] Status In-Reply-To: References: <432fb3ea-cbed-2579-5721-26f67b502f93@canterbury.ac.nz> <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> Message-ID: <69B56638-F69D-4499-B113-46F903B98068@internode.on.net> > On 1 Feb 2020, at 09:49, Greg Ewing wrote: > > On 1/02/20 3:03 am, John Skaller2 wrote: >> So this is some kind of hack way >> of getting something a bit like Haskell type classes, >> you?re basically saying int32_t and int64_t are of class ?Integer?. > > I suppose you could think of it that way, but it's really > not that formal. > >> This also explains the conflict for me, because Felix is the opposite: >> it aims to make the types of things more precise (and has actual >> type classes for generalisation). > > To define them any more precisely, Cython would need to > know how things vary depending on the platform, which would > mean conditional compilation, etc. It's much easier to leave > all that up to the C compiler and system headers. It also > ensures that there can't be any mismatch between the two. But the all hell breaks loose for pointers. Your hack only works for rvalues. Of course you probably know this doesn?t occur. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Fri Jan 31 21:42:21 2020 From: skaller at internode.on.net (John Skaller2) Date: Sat, 1 Feb 2020 13:42:21 +1100 Subject: [Cython] Size of output In-Reply-To: References: <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> Message-ID: > >> Is there an option to use an #include for the standard stuff? > > There are upsides and downsides to that as well. Hence an option. But it could be work to implement so I?m just exploring at the moment. > The way > things are, the generated file is self-contained, and can > be shipped without worrying about it becoming disconnected > from a compatible version of the include file. This is > important when details of the support code can change > without notice between Cython releases. Yes, and in this case an include file may actually be better because it will upgrade with Cython. YMMV I guess. But the main reason is to remove a lot of useless clutter. ? John Skaller skaller at internode.on.net From skaller at internode.on.net Fri Jan 31 22:40:05 2020 From: skaller at internode.on.net (John Skaller2) Date: Sat, 1 Feb 2020 14:40:05 +1100 Subject: [Cython] Size of output In-Reply-To: References: <1CE7B193-D458-4C9F-8136-1D7645692DEB@internode.on.net> <9F9FA658-CA8E-4DB2-BC62-3D978D54B3A9@internode.on.net> <6E79F14A-BF42-4AA8-A030-5E5797D43765@internode.on.net> <4c347e76-3f4c-cb24-27f3-c6fc1a11cbd1@canterbury.ac.nz> <8D1932F6-3858-4F23-A075-5DF96B94C0E8@internode.on.net> <8F478954-6F00-4A14-A641-0D654FE60E24@internode.on.net> <07D8E733-EB80-4FA9-B0A0-0BD2ED744FE5@internode.on.net> <2129c6a8-add5-2a6c-8c10-40a3cdc3c2db@canterbury.ac.nz> <435F1F1C-5A6E-4607-89A3-9132E858E8B0@internode.on.net> Message-ID: <6EBDBA6D-4C98-4DF2-AC00-609A5E462635@internode.on.net> > I agree there's some fat that could be trimmed there, but not sure > it'd be worth the effort. You?re probably right. Its a problem writing a compiler in a language wholy unsuited for the job. Even with a more suitable language, emitting code, in the right order, with just the things actually required, is difficult. I use a multi-pass predictive system and a multi-pass code generator and I find bugs all the time because it isn?t run by actually dependencies but predicted ones. ? John Skaller skaller at internode.on.net