From lanyjie at yahoo.com Wed Aug 5 20:10:29 2009 From: lanyjie at yahoo.com (Yingjie Lan) Date: Wed, 5 Aug 2009 11:10:29 -0700 (PDT) Subject: [capi-sig] Reasons for using expy In-Reply-To: <4A797106.8040807@gmail.com> Message-ID: <363186.73070.qm@web54203.mail.re2.yahoo.com> Hi, The expy project provides an express way to extend Python. After some careful considerations, I came up with some reasons for expy (this is not an exhaustive list): (I). WYSIWYG. The expy project enables you to write your module in Python the way your extension would be (WYSIWYG), and meanwhile write your implementation in pure C. You specify your modules, functions, methods, classes, and even their documentations the usual way of writing your Python correspondences. Then your provide your implementation to the functions/methods by returning a multi-line string. By such an arrangement, everything falls in its right place, and your extension code becomes easy to read and maintain. Also, the generated code is very human-friendly. (II). You only provide minimal information to indicate your intension of how your module/class would function in Python. So your extension is largely independent from the Python extension API. As your interaction with the Python extension API is reduced to minimal (you only care about the functionality and logic), it is then possible that your module written in expy can be independent of changes in the extension API. (III). The building and setup of your project can be automatically done with the distutil tool. In the tutorial, there are ample examples on how easily this is achieved. (IV). Very light weight. The expy tool is surprisingly light weight dispite of its powerful ability, as it is written in pure Python. There is no parser or compiler for code generation, but rather the powerful reflexion mechanism of Python is exploited in a clever way to generate human-friendly codes. Currently, generating code in C is supported, however, the implementation is well modularized and code generation in other languages such as Java and C++ should be easy. While there are already a couple of other projects trying to simply this task with different strategies, such as Cython, Pyrex and modulator, this project is unique and charming in its own way. All you need is the WYSIWYG Python file for your module extension, then expy takes care of everything else. What follows in this documentation is on how to extend Python in C using expy-cxpy: the module expy helps define your module, while module cxpy helps generate C codes for your defined module. For more information about expy, please visit its homepage at: http://expy.sf.net/ Cheers, Yingjie From lanyjie at yahoo.com Wed Aug 12 06:15:39 2009 From: lanyjie at yahoo.com (Yingjie Lan) Date: Tue, 11 Aug 2009 21:15:39 -0700 (PDT) Subject: [capi-sig] releasing expy 0.1.1 Message-ID: <932447.5519.qm@web54201.mail.re2.yahoo.com> Hi all, This is to announce the release of expy 0.1.1 for those who are interested. Thanks a lot for the interest from this list. For more information, please see http://expy.sf.net/ Note: expy is an express way to extend Python. Why consider expy? Here are some good reasons: (I). WYSIWYG. The expy project enables you to write your module in Python the way your extension would be (WYSIWYG), and meanwhile write your implementation in pure C. You specify your modules, functions, methods, classes, and even their documentations the usual way of writing your Python correspondences. Then your provide your implementation to the functions/methods by returning a multi-line string. By such an arrangement, everything falls in its right place, and your extension code becomes easy to read and maintain. Also, the generated code is very human-friendly. (II). You only provide minimal information to indicate your intension of how your module/class would function in Python. So your extension is largely independent from the Python extension API. As your interaction with the Python extension API is reduced to minimal (you only care about the functionality and logic), it is then possible that your module written in expy can be independent of changes in the extension API. (III). The building and setup of your project can be automatically done with the distutil tool. In the tutorial, there are ample examples on how easily this is achieved. (IV). Very light weight. The expy tool is surprisingly light weight dispite of its powerful ability, as it is written in pure Python. There is no parser or compiler for code generation, but rather the powerful reflexion mechanism of Python is exploited in a clever way to generate human-friendly codes. Currently, generating code in C is supported, however, the implementation is well modularized and code generation in other languages such as Java and C++ should be easy. While there are already a couple of other projects trying to simply this task with different strategies, such as Cython, Pyrex and modulator, this project is unique and charming in its own way. All you need is the WYSIWYG Python file for your module extension, then expy takes care of everything else. What follows in this documentation is on how to extend Python in C using expy-cxpy: the module expy helps define your module, while module cxpy helps generate C codes for your defined module. regards, Yingjie From jerome.fuselier at free.fr Wed Aug 12 18:29:48 2009 From: jerome.fuselier at free.fr (=?ISO-8859-1?Q?J=E9r=F4me_Fuselier?=) Date: Wed, 12 Aug 2009 17:29:48 +0100 Subject: [capi-sig] Naive question on references Message-ID: <4A82EDFC.70107@free.fr> Hello, I have a working application which extends a C api to python and everything works as I wanted to. I'm now in a cleaning phase where I want to improve my stuff, especially to avoid memory leak. Here is a simple example of what seems correct to me but for which I would like an expert opinion : PyObject * test() { PyObject *dict = PyDict_New(); // ... some code to initialize dict PyObject *item = PyDict_GetItemString(dict, "key"); Py_INCREF(item); Py_DECREF(dict); return item; } As far as I understood, PyDict_GetItemString() returns a borrowed reference to the element so I need to do Py_INCREF(item) to own this reference. But is this needed ? If I only do return item, will the received item be owned by the caller of the test method ? Thanks, Jerome From gjcarneiro at gmail.com Thu Aug 13 00:46:15 2009 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Wed, 12 Aug 2009 23:46:15 +0100 Subject: [capi-sig] Naive question on references In-Reply-To: <4A82EDFC.70107@free.fr> References: <4A82EDFC.70107@free.fr> Message-ID: 2009/8/12 J?r?me Fuselier > Hello, > I have a working application which extends a C api to python and > everything works as I wanted to. I'm now in a cleaning phase where I want to > improve my stuff, especially to avoid memory leak. > > Here is a simple example of what seems correct to me but for which I would > like an expert opinion : > > PyObject * test() { > PyObject *dict = PyDict_New(); > // ... some code to initialize dict > > PyObject *item = PyDict_GetItemString(dict, "key"); > Py_INCREF(item); > Py_DECREF(dict); > return item; > } > > As far as I understood, PyDict_GetItemString() returns a borrowed reference > to the element so I need to do Py_INCREF(item) to own this reference. But is > this needed ? If I only do return item, will the received item be owned by > the caller of the test method ? Yes, you do need the INCREF. If you remove the Py_INCREF(item) line, then when Py_DECREF(dict) is executed the dictionary could be freed, along with the object pointed to by 'item'. The code above is correct, and the caller of your test() function is responsible for DECREF'ing the returned value. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert From gjcarneiro at gmail.com Thu Aug 13 10:49:04 2009 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Thu, 13 Aug 2009 09:49:04 +0100 Subject: [capi-sig] Naive question on references In-Reply-To: <4A83D1DC.7070406@free.fr> References: <4A82EDFC.70107@free.fr> <4A83D1DC.7070406@free.fr> Message-ID: 2009/8/13 J?r?me Fuselier > > > Gustavo Carneiro wrote: > > > > 2009/8/12 J?r?me Fuselier > >> Hello, >> I have a working application which extends a C api to python and >> everything works as I wanted to. I'm now in a cleaning phase where I want to >> improve my stuff, especially to avoid memory leak. >> >> Here is a simple example of what seems correct to me but for which I would >> like an expert opinion : >> >> PyObject * test() { >> PyObject *dict = PyDict_New(); >> // ... some code to initialize dict >> >> PyObject *item = PyDict_GetItemString(dict, "key"); >> Py_INCREF(item); >> Py_DECREF(dict); >> return item; >> } >> >> As far as I understood, PyDict_GetItemString() returns a borrowed >> reference to the element so I need to do Py_INCREF(item) to own this >> reference. But is this needed ? If I only do return item, will the received >> item be owned by the caller of the test method ? > > > Yes, you do need the INCREF. If you remove the Py_INCREF(item) line, then > when Py_DECREF(dict) is executed the dictionary could be freed, along with > the object pointed to by 'item'. > > The code above is correct, and the caller of your test() function is > responsible for DECREF'ing the returned value. > > > Thanks, > > That's confirm my understanding so that's fine. > > I have another question related to memory management in the python api. > > I often see this code : > > char *s; > ok = PyArg_ParseTuple(args, "s", &s); > .. > > But I never saw if there's a need/way to free the memory for s. I tried > with malloc and PyMem_Free but this leads to segmentation fault. I often > have large data passed as parameters, with a process that can last long so I > would like to ensure that there's not a memory leak in that part. > No need to free memory for 's'. The string pointer stored in 's' is valid as long as the 'args' reference is valid, but it is a borrowed pointer, bound to the life cycle of 'args'. That is, you can do what you want with 's' but don't keep this pointer beyond the scope where 'args' is defined. If you really need the string beyond this scope, strdup() is your friend. -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert From lanyjie at yahoo.com Fri Aug 14 19:00:49 2009 From: lanyjie at yahoo.com (Yingjie Lan) Date: Fri, 14 Aug 2009 10:00:49 -0700 (PDT) Subject: [capi-sig] [Python-Dev] expy: an expressway to extend Python In-Reply-To: Message-ID: <637900.988.qm@web54204.mail.re2.yahoo.com> --- On Sat, 8/8/09, Stefan Behnel wrote: > From: Stefan Behnel > Subject: Re: [Python-Dev] expy: an expressway to extend Python > To: python-dev at python.org > Date: Saturday, August 8, 2009, 4:55 PM > > More details at http://expy.sourceforge.net/ > > I'm clearly biased, but my main concern here is that expy > requires C code > to be written inside of strings. There isn't any good > editor support for > that, so I doubt that expy is good for anything but very > thin wrappers (as > in the examples you presented). Thanks a lot for the input -- I sort of recaptured the advantages of expy and listed four points in the new introduction at http://expy.sf.net/ homepage. Lacking of editor highlight support is quite a problem, but it is possible to create a support. For example, you can use this to indicate the start of embedded code highlight: return """ and then the end mark is of course the enclosing """ > > That said, you might want to look at the argument unpacking > code generated > by Cython. It's highly optimised through specialisation and > has been > benchmarked quite a bit faster than the generic Python > C-API functions for > tuple/keyword extracting. Since argument conversion seems > to be more or > less all that expy really does, maybe you want to reuse > that code. > > Stefan Oh sure, that's nice if that part can be adopted by expy-cxpy. Any help out on this would be very welcomed. Yingjie