From cbarton at metavr.com Mon Jul 2 15:30:57 2007 From: cbarton at metavr.com (Campbell Barton) Date: Mon, 02 Jul 2007 23:30:57 +1000 Subject: [capi-sig] Documenting a C/Py API Message-ID: <4688FE11.6040407@metavr.com> Hi Guys, first post on the list, dont be shy ;) In Blender3D were doing an API refactor and Im looking at better ways to write the Python C/API documentation. At the moment we use epydoc, Its all Iv used so I don't have much to compare it to. If you havnt used it, its basically a way to add formatting to your normal .py docstrings so they can be converted into HTML or PDF. This works well with python only projects but is not somthing you can do in C without some manual docstring extraction from the C/API. for blender3d we have C doctrings as well as python files that just contain functions with docstrings (no actual code) for epydoc to write them to html. The C/API docstings often end up being brief and not that helpful, since the epydocs are what most people use so more effort is put there. Here are the current epydocs. http://www.blender.org/documentation/244PythonDoc/index.html Ideally Id like to only maintain 1 set of docs... so possibilities are 1) dont write epydocs, only use docstrings (we need a way we can browse them as HTML still) 2) dont write C API/docstrings, just have empty strings where all the docstrings would go. 3) include epydoc formatting in C API/docstrings and pass these docstrings to epydoc for it to generate HTML. Iv got 3) working as a testcase, the message attached details how this works with the pro's and con's of adding epydoc tags into C/API docstrings. However Id be interested to know how other projects deal with the issue of maintaining docstrings and online docs. Cheers From cbarton at metavr.com Mon Jul 2 16:12:46 2007 From: cbarton at metavr.com (Campbell Barton) Date: Tue, 03 Jul 2007 00:12:46 +1000 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <4688FE11.6040407@metavr.com> References: <4688FE11.6040407@metavr.com> Message-ID: <468907DE.3020400@metavr.com> grrr my attachment was stripped here it is inline.. Campbell Barton wrote: > Campbell Barton wrote: >> Revision: 11145 >> http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=11145 >> Author: campbellbarton >> Date: 2007-07-02 14:22:30 +0200 (Mon, 02 Jul 2007) >> >> Log Message: >> ----------- >> Testing a way to have python docs in Blenders PyAPI docstrings. >> rather then maintaining 2 sets of docs which is what we have done for a long time. >> >> Works for Group and Camera now, here are the results >> http://members.optusnet.com.au/cjbarton/bpy_docs/camera.Camera-class.html >> http://members.optusnet.com.au/cjbarton/bpy_docs/group.Group-class.html >> >> This means the C docstrings need to contain epydoc formatting (humanly readable) but need to be changed in the C files. though editing be done in py files and the final version copied back into the .c file. >> >> _epy_write_docstrings.py - extracts epydocs and examples into their own py files. >> _epy_docgen.sh - runs _epy_write_docstrings.py in background mode and then epydoc to make the html docs. >> >> advantage is less work with duplicate docs, we can automate running all examples within the docs to make sure they work. > > Hi, just committed an experimental C docstring/epydoc metrhod of > maintaining our python API docs. its worth discussing the options for docs. > > this is not ideal (epydocs in C are a bit of a kludge) but it as some > good points and IMHO is better then what we have been doing. > > pros: > * 1 set of docs, not 2 > * built-in python methods are added, like getitem or __hash__ > * we can do preprocessing since the epydocs are made from a script... > * better C docstrings (albeit with a bit of odd epydoc tags) - people do > use the C docstrings sometimes and at the moment they are not well > checked - some have mistakes or are quite uninformative. > * extracts examples to py files so we can automate running them all. > * its more flexible, we could for instance have all examples in py files > and include them when the blender puts the docs together, this would > also not bloat blender too much with really big docstrings, > * converting existing epydocs into C/epy/docstrings is not hard. > > cons: > * editing large docstrings with \n\ at the end of everyline is not nice. > * nonstandard, uses a custom script (~200 lines) to extract and write > docstrings > * workflow to writing docs is longer since you need to edit the c file, > compile blender and run the script. If there is a missing tab youd not > want to do all that again. > ... Realisticly its not so bad since you can edit the docs that blender > spits out and then update the C file once your done, but the process is > still longer for writing epydocs though arguably better then editing > docs in 2 places. > > * one more hoop to jump through when writing docs - you have to learn > our way of adding epydocs in C. > > - Even though Id rather not make it harder for new people to come in > and write docs, so far python scripters have not come up and offered to > write docs, instead its been the C/API authors - who are alredy familiar > with the C files, it also seems that very few people besides the API > devs even use epydoc or have it installed so Im not too worried about this. > > Have a look at Camera.c to see what the epy/docstrings look like. > > http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/Group.c?view=markup&root=bf-blender&pathrev=11145 > > If we deciede this is acceptable I can continue and move other finished > areas of the new API to epy/docstrings. > > > _______________________________________________ > Bf-python mailing list > Bf-python at blender.org > http://lists.blender.org/mailman/listinfo/bf-python > From carsten at uniqsys.com Mon Jul 2 16:22:05 2007 From: carsten at uniqsys.com (Carsten Haese) Date: Mon, 02 Jul 2007 10:22:05 -0400 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <4688FE11.6040407@metavr.com> References: <4688FE11.6040407@metavr.com> Message-ID: <1183386125.3415.7.camel@dot.uniqsys.com> On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton wrote: > However Id be interested to know how other projects deal with the issue > of maintaining docstrings and online docs. I maintain separate docstrings and online docs for one simple reason: The online docs can be more extensive than docstrings. Docstrings are for brief memory-joggers. Online documentation can contain examples and generally more detail that's not appropriate for docstrings. Just my two cents, -- Carsten Haese http://informixdb.sourceforge.net From theeth at yahoo.com Mon Jul 2 17:46:42 2007 From: theeth at yahoo.com (Martin Poirier) Date: Mon, 2 Jul 2007 08:46:42 -0700 (PDT) Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <1183386125.3415.7.camel@dot.uniqsys.com> Message-ID: <561833.66458.qm@web51309.mail.re2.yahoo.com> --- Carsten Haese wrote: > On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton > wrote: > > However Id be interested to know how other > projects deal with the issue > > of maintaining docstrings and online docs. > > I maintain separate docstrings and online docs for > one simple reason: > The online docs can be more extensive than > docstrings. Docstrings are > for brief memory-joggers. Online documentation can > contain examples and > generally more detail that's not appropriate for > docstrings. Can you elaborate on this a bit. What kind of tools do you use to generate/maintain your docs? Martin - Blender developer ____________________________________________________________________________________ Finding fabulous fares is fun. Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains. http://farechase.yahoo.com/promo-generic-14795097 From barry at python.org Mon Jul 2 19:12:40 2007 From: barry at python.org (Barry Warsaw) Date: Mon, 2 Jul 2007 13:12:40 -0400 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <4688FE11.6040407@metavr.com> References: <4688FE11.6040407@metavr.com> Message-ID: <3D87E675-B27A-4072-B071-AC20E690170F@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 2, 2007, at 9:30 AM, Campbell Barton wrote: > At the moment we use epydoc, Its all Iv used so I don't have much > to compare it to. > > If you havnt used it, its basically a way to add formatting to your > normal .py docstrings so they can be converted into HTML or PDF. > > This works well with python only projects but is not somthing you > can do in C without some manual docstring extraction from the C/API. This isn't necessarily true. In a previous life, I was doing tons of C API stuff, essentially building and maintaining a big embedded/ extended Python application. We docstring'd all of our C API and used epydoc to generate our API documentation. It worked very well, with a few caveats. First, epydoc had some problems in earlier versions, for example, the fact that a class's docstring and its constructor's docstring both live in the same slot in C. IIRC, epydoc also had some problems with certain C inheritance style we were using, and a bunch of other minor problems. AFAIK, these have mostly been addressed with newer epydoc versions, though I honestly haven't looked at it for 6+ months. Second, and more importantly, because of the nature of our app, we had to run epydoc programmatically, inside the application's shell. I think you're forced to use epydoc's import-based discovery mechanisms when docstring'ing C code, so your modules have to be importable. Because we were an embedded application, the package namespace for our C code wasn't available except when the application shell was run. Therefore we added a command line option that would run the epydoc stuff, once Python was initialized and the extension modules were available. Other than that, we really didn't have to do much special to docstring our C functions, provide them in normal Python interactive interpreter help, and have nicely generated epydoc documentation. All of our docstrings lived right in C and were attached to the C functions using the normal Python C API structures. I wish I could post some examples, but the code was closed-source and the company no longer exists. ;) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRokyCXEjvBPtnXfVAQIYhwP/R/y9Jq18PaQzOkhDBdjHKsxo0bOx3FUN OFTYsDarPHpuxUVSRTqswjhi1Q/izP+NKcKeRRNA2KLqyrsMcVi0iWcMiqqZmosB ZZqSTmTgCXRqjoEZuCBHOb94YCYyn8J93ein95W0E2enUNVmYfNoL67/uqSVlUm1 wHS37QyBowQ= =UrSn -----END PGP SIGNATURE----- From carsten at uniqsys.com Mon Jul 2 20:36:58 2007 From: carsten at uniqsys.com (Carsten Haese) Date: Mon, 02 Jul 2007 14:36:58 -0400 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <561833.66458.qm@web51309.mail.re2.yahoo.com> References: <561833.66458.qm@web51309.mail.re2.yahoo.com> Message-ID: <1183401418.3415.19.camel@dot.uniqsys.com> On Mon, 2007-07-02 at 08:46 -0700, Martin Poirier wrote: > --- Carsten Haese wrote: > > > On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton > > wrote: > > > However Id be interested to know how other > > projects deal with the issue > > > of maintaining docstrings and online docs. > > > > I maintain separate docstrings and online docs for > > one simple reason: > > The online docs can be more extensive than > > docstrings. Docstrings are > > for brief memory-joggers. Online documentation can > > contain examples and > > generally more detail that's not appropriate for > > docstrings. > > Can you elaborate on this a bit. What kind of tools do > you use to generate/maintain your docs? My needs are probably significantly different, i.e. smaller, than yours. My documentation is all in one moderately sized Restructured Text file. If you were expecting automatically generated hyperlinked API docs, you'll be disappointed. I was merely making the point that in my opinion it's not a problem to have two sets of documentation because docstrings and online documentation serve different purposes. Your opinion may vary. -- Carsten Haese http://informixdb.sourceforge.net From cbarton at metavr.com Tue Jul 3 04:14:50 2007 From: cbarton at metavr.com (Campbell Barton) Date: Tue, 03 Jul 2007 12:14:50 +1000 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <561833.66458.qm@web51309.mail.re2.yahoo.com> References: <561833.66458.qm@web51309.mail.re2.yahoo.com> Message-ID: <4689B11A.9010803@metavr.com> Martin Poirier wrote: > --- Carsten Haese wrote: > >> On Mon, 2007-07-02 at 23:30 +1000, Campbell Barton >> wrote: >>> However Id be interested to know how other >> projects deal with the issue >>> of maintaining docstrings and online docs. >> I maintain separate docstrings and online docs for >> one simple reason: >> The online docs can be more extensive than >> docstrings. Docstrings are >> for brief memory-joggers. Online documentation can >> contain examples and >> generally more detail that's not appropriate for >> docstrings. > > Can you elaborate on this a bit. What kind of tools do > you use to generate/maintain your docs? > > Martin > > - Blender developer Its just one script that runs inside blender and writes its docstrings to py files epydoc can use. http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/doc/_epy_write_docstrings.py?view=markup&root=bf-blender&pathrev=11145 At the moment it just takes a PyType, and writes a file with its methods and attributes. adding indenting and method names to the epydocs. I wrote this last night and it only works with the Group and Camera types, so if we choose to keep using this method we can extend and modify as needed. Barry pointed out that docs could be generated with epydoc inspecting the C api directly, Id be interested to know if you had much epydoc spesific stuff in these docstrings, though i was aware this was possible in some cases, you end up with nicer docs if they include formatting tags. Carsten, agree Examples within the C docstrings is bloating them too much, Id like to add these in when generating the py files with some form of #include that inserts an existing Py file. Thanks for your feedback - I may be over reacting about maintaining 2 sets of docstrings because when I first started with Blenders Python api there would be for instance.. mesh.getName() mesh.setName() mesh.name And each needed a docstring and epydoc, so we had to have 6 different bits of documentation for each setting. We have decided not to use get/set style anymore so its only 2 - which may be a necessary annoyance and not that bad in practice. This is the biggest PyType we have, to get an idea of what Im talking about... file:///cvs_blender/blender/source/blender/python/api2_2x/doc/BPY_API/Object.Object-class.html From cbarton at metavr.com Tue Jul 3 04:53:07 2007 From: cbarton at metavr.com (Campbell Barton) Date: Tue, 03 Jul 2007 12:53:07 +1000 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <4689B11A.9010803@metavr.com> References: <561833.66458.qm@web51309.mail.re2.yahoo.com> <4689B11A.9010803@metavr.com> Message-ID: <4689BA13.2040701@metavr.com> Campbell Barton wrote: > This is the biggest PyType we have, to get an idea of what Im talking > about... > > file:///cvs_blender/blender/source/blender/python/api2_2x/doc/BPY_API/Object.Object-class.html bugger... http://members.optusnet.com.au/cjbarton/BPY_API/Object.Object-class.html From barry at python.org Wed Jul 4 01:49:55 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 3 Jul 2007 19:49:55 -0400 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <4689B11A.9010803@metavr.com> References: <561833.66458.qm@web51309.mail.re2.yahoo.com> <4689B11A.9010803@metavr.com> Message-ID: <3509529D-54AD-41A6-BCE3-87DCA9FE5659@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 2, 2007, at 10:14 PM, Campbell Barton wrote: > Barry pointed out that docs could be generated with epydoc inspecting > the C api directly, > Id be interested to know if you had much epydoc spesific stuff in > these > docstrings, though i was aware this was possible in some cases, you > end > up with nicer docs if they include formatting tags. I'm not sure if you were asking me this question, but I'll answer anyway :). Yes! We put epydoc markup in our C API docstrings and epydoc almost all the right things. As I mentioned there were a couple of quirks due to import weirdness or inheritance but 98% of the docstrings worked great. I should note one other bit of funny business, Writing multiline docstrings in C is a royal PITA of course, especially if you ever have to re-fill a paragraph. Ain't no TQS's there! The one thing that bit us fairly often was a limit on the number of characters inside a string literal that Microsoft's compiler imposed on us. Our code was portable between gcc and MSVC (Studio Enterprise .NET Ultramarine Whizzy XP Light and Happiness Edition 2003 (tm)(r)(c) (blah)). gcc never had a problem but with MSVC we couldn't use the \n \ trick after 1024 characters. We'd have to close the literal and start again. There was probably some magic config buried under 30,000 menu options, but we never found it. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRorgpHEjvBPtnXfVAQLWnQQAhU+JslzLZYLeXEMl5/cSjugAgU1YJNdx 2xzp4qK1Oi/RYgRReoCBEvbNA/9+h3OARda+Uj9GiBhTEZziC+M3uqTqEFltFIt2 zVXKF9iHB4jNwiuu0L5e/hA3CxkFOKFCypt8JLzi9nJzOy3iTQv+KYqFfM8lMZvu hzheqIubQ6I= =WayZ -----END PGP SIGNATURE----- From snakeru at gmail.com Tue Jul 3 21:09:29 2007 From: snakeru at gmail.com (Alexey Nezhdanov) Date: Tue, 3 Jul 2007 23:09:29 +0400 Subject: [capi-sig] is deinitialisation of embedded python equal to full restart? Message-ID: <200707032309.29935.snakeru@gmail.com> Hello. Sorry if posting to wrong maillist, I have read full list and decided that this one is the most appropriate. I have a quick question: am I able to do multiple initialisations/deinitialisations of embedded python interpreter? I mean - I'd like to init python, run a script in it for some time (say, couple of hours) and then just deinitialise python so everything, including the python program that didn't expected such brutal end to be freed and deinitialised. Then, after some more time, call Py_Initialize() again and again load and start using my python program. The most essential for me is freeing all sorts of RAM that python allocated for objects or whatever else. IOW - I want python to be returned in virgin state as it was on my program load w/o exiting my program and closing all files/tcp connections that it have opened/or resetting any of other my C data structures that were created. Is that possible? -- Respectfully Alexey Nezhdanov From hniksic at xemacs.org Wed Jul 4 10:32:31 2007 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Wed, 04 Jul 2007 10:32:31 +0200 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <4688FE11.6040407@metavr.com> (Campbell Barton's message of"Mon, 02 Jul 2007 23:30:57 +1000") References: <4688FE11.6040407@metavr.com> Message-ID: <87abucwov4.fsf@mulj.homelinux.net> Campbell Barton writes: > Ideally Id like to only maintain 1 set of docs... so possibilities > are [...] If you maintain only one set of docs, then your options are indeed limited. I tend to agree with Carsten that docstrings and full documentation are in fact different; docstrings tend to be briefer, don't provide examples, and in fact don't have real formatting. Attempts to add HTML-like formatting to docstrings lead to them not being readable from Python, which defeats the purpose. The Python source is a precedent for maintaining two sets of documentation and for the distinction between them. We currently document the C-implemented classes with Doxygen by writing dummy classes with markup that Doxygen understands. The line numbers are off and Doxygen doesn't understand some Python features (and tries to shoehorn others into C++ terminology), but other than that, it serves us well. We haven't tried epydoc yet. From cbarton at metavr.com Wed Jul 4 18:00:43 2007 From: cbarton at metavr.com (Campbell Barton) Date: Thu, 05 Jul 2007 02:00:43 +1000 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <3509529D-54AD-41A6-BCE3-87DCA9FE5659@python.org> References: <561833.66458.qm@web51309.mail.re2.yahoo.com> <4689B11A.9010803@metavr.com> <3509529D-54AD-41A6-BCE3-87DCA9FE5659@python.org> Message-ID: <468BC42B.5000805@metavr.com> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Jul 2, 2007, at 10:14 PM, Campbell Barton wrote: > >> Barry pointed out that docs could be generated with epydoc inspecting >> the C api directly, >> Id be interested to know if you had much epydoc spesific stuff in these >> docstrings, though i was aware this was possible in some cases, you end >> up with nicer docs if they include formatting tags. > > I'm not sure if you were asking me this question, but I'll answer anyway > :). Yes! We put epydoc markup in our C API docstrings and epydoc > almost all the right things. As I mentioned there were a couple of > quirks due to import weirdness or inheritance but 98% of the docstrings > worked great. > > I should note one other bit of funny business, Writing multiline > docstrings in C is a royal PITA of course, especially if you ever have > to re-fill a paragraph. Ain't no TQS's there! The one thing that bit > us fairly often was a limit on the number of characters inside a string > literal that Microsoft's compiler imposed on us. Our code was portable > between gcc and MSVC (Studio Enterprise .NET Ultramarine Whizzy XP Light > and Happiness Edition 2003 (tm)(r)(c)(blah)). gcc never had a problem > but with MSVC we couldn't use the \n\ trick after 1024 characters. We'd > have to close the literal and start again. There was probably some > magic config buried under 30,000 menu options, but we never found it. > > - -Barry For people who dont mind docstrings + online docs, how big are your projects? I just did a quick check on Blender/Python epydocs and there are 1297 functions and 868 attributes and 73 classes == ~2238 doc strings. (got my numbers from running this in the epydoc dir) cat *.py | grep "def " | grep ":" | wc -l cat *.py | grep "class " | grep ":" | wc -l cat *.py | grep "@ivar" | wc -l Barry: Looks like this is a workable solution that Ill go ahead with. I was hoping there was some kind of triple quoting in C :/ \n\ at the end of every line is a PITA, Though with search replace it can be added after writing.. - Thanks for the hint regarding 1024 chars. EpyDocs as C strings aren't so bad, heres an example (for people other then Barry) of how it looks for getseters. {"orthographic", (getter)Camera_getOrtho, (setter)Camera_setOrtho, "\ @ivar orthographic: When true this camera will render with no perspective.\n\ @type orthographic: bool", NULL}, {"ipo", (getter)Camera_getIpo, (setter)Camera_setIpo, "\ @ivar ipo: The ipo linked to this camera data object.\n\ Assign None to clear the ipo from this camera.\n\ @type ipo: Blender Ipo", NULL}, adding some special tag like ((reference some_example.py)) that can be inserted when generating the Py files is no big deal and would read well in the docstrings too. Thanks again. - Cam From barry at python.org Thu Jul 5 14:49:21 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 5 Jul 2007 08:49:21 -0400 Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <468BC42B.5000805@metavr.com> References: <561833.66458.qm@web51309.mail.re2.yahoo.com> <4689B11A.9010803@metavr.com> <3509529D-54AD-41A6-BCE3-87DCA9FE5659@python.org> <468BC42B.5000805@metavr.com> Message-ID: <41E7FB24-4ED4-4366-AF38-4FC2747DFB4A@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 4, 2007, at 12:00 PM, Campbell Barton wrote: > For people who dont mind docstrings + online docs, how big are your > projects? Our project was pretty big. I don't have exact numbers any more, but I'd say on the order of maybe two dozen C implemented classes with several hundred methods and properties in total. > I just did a quick check on Blender/Python epydocs and there are 1297 > functions and 868 attributes and 73 classes == ~2238 doc strings. I'm not counting stuff implemented in Python, which was probably 2-5 times the size of our C stuff. OTOH, our C classes were the core of our model so we took great pains to document them accurately. > Barry: > Looks like this is a workable solution that Ill go ahead with. > > I was hoping there was some kind of triple quoting in C :/ > \n\ at the end of every line is a PITA, Though with search > replace it > can be added after writing.. > > - Thanks for the hint regarding 1024 chars. > > > EpyDocs as C strings aren't so bad, heres an example (for people other > then Barry) of how it looks for getseters. > > {"orthographic", > (getter)Camera_getOrtho, (setter)Camera_setOrtho, "\ > @ivar orthographic: When true this camera will render with no > perspective.\n\ > @type orthographic: bool", > NULL}, > {"ipo", > (getter)Camera_getIpo, (setter)Camera_setIpo, "\ > @ivar ipo: The ipo linked to this camera data object.\n\ > Assign None to clear the ipo from this camera.\n\ > @type ipo: Blender Ipo", > NULL}, Just to relate my own experiences here. I would typically include a PyDoc_STRVAR() call right before the function's static definition, instead of including the docstring in the structure definition. I found that was cleaner and made it easier to update the docstring when the function was updated. YMMV of course. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRozo0XEjvBPtnXfVAQKkewQAinerU5Jc+jyMWD/6Drkx/3p7t9Z+MfFh 0t3NJt5fbY0VXJRh2R4ghApJpWXXNp1BQ3qy6YnRTb5zNhT+dz6V5iipenLdWyYg ShE5MoHdhEPrEZlQMxgDzJNHBiAYtDelGdyyJEwspgmE6expbt8z1s6QvML2dGgQ u147d9X2geo= =VYpg -----END PGP SIGNATURE----- From e-huss at netmeridian.com Tue Jul 3 23:54:43 2007 From: e-huss at netmeridian.com (Eric Huss) Date: Tue, 3 Jul 2007 14:54:43 -0700 (PDT) Subject: [capi-sig] Documenting a C/Py API In-Reply-To: <4688FE11.6040407@metavr.com> References: <4688FE11.6040407@metavr.com> Message-ID: > In Blender3D were doing an API refactor and Im looking at better ways to > write the Python C/API documentation. > > At the moment we use epydoc, Its all Iv used so I don't have much to > compare it to. > > If you havnt used it, its basically a way to add formatting to your > normal .py docstrings so they can be converted into HTML or PDF. > > This works well with python only projects but is not somthing you can do > in C without some manual docstring extraction from the C/API. I use epydoc extensively, with a lot of C code. Actually, a lot of it is Pyrex, so it's much easier to write and is more natural, but the basics should be the same. The latest versions of epydoc are much better. I would suggest getting the latest straight from subversion, it is pretty stable. I think Barry may have mentioned this, but you can put __init__ signatures in the type docstring. I haven't experimented with this, much. This isn't completely documented, you can look in epydoc/docstringparser.py and look for SIGNATURE_RE. See this message from some detail: http://sourceforge.net/mailarchive/message.php?msg_name=8D22D41A-730E-42B4-A583-FBF3D93C59FB%40gradient.cis.upenn.edu I have one of my projects up where you can look at the docs generated by epydoc. The majority of this code is in Pyrex: http://ehuss.org/mysql/api/ -Eric From cbarton at metavr.com Wed Jul 11 12:58:19 2007 From: cbarton at metavr.com (Campbell Barton) Date: Wed, 11 Jul 2007 20:58:19 +1000 Subject: [capi-sig] Getting Numbers in C Message-ID: <4694B7CB.5080802@metavr.com> Iv noticed that theres no one liner way to get a number (double) from a PyNumber from the C api. PyFloat_AsDouble - http://www.python.org/doc/api/floatObjects.html - Does what I want, but only works on PyFloat (and PyInts), not user implementations of the PyNumber protocol. PyNumber_Float - http://www.python.org/doc/api/number.html - does what I need but, returnd a PyFloat that needs to then be passed to PyFloat_AsDouble, PyNumber_Float Id like somthing like PyNumber_AsDouble, thinking of writing a utility function for this that could read... /* assumes PyNumber_Check has been tested, should not segfault either way */ double PyNumber_AsDouble(PyObject * value) { if (PyFloat_Check(value) || PyInt_Check(value)) { return PyFloat_AsDouble(value); } else { PyObject *pyfloat; double d; pyfloat = PyNumber_Float(value); d = PyNumber_AsDouble(pyfloat); Py_XDECREF(pyfloat); return d; } } Using PyNumber_Float for all values would be a bit slower since it needs to make a new pyfloat each time which is why Id prefer to use PyFloat_AsDouble where possible. Wondering if other people would fine something like this useful in the Py/C api - or if Im missing something. Thanks -- Campbell J Barton (ideasman42) From hniksic at xemacs.org Fri Jul 13 12:29:48 2007 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Fri, 13 Jul 2007 12:29:48 +0200 Subject: [capi-sig] is deinitialisation of embedded python equal to fullrestart? In-Reply-To: <200707032309.29935.snakeru@gmail.com> (Alexey Nezhdanov'smessage of "Tue, 3 Jul 2007 23:09:29 +0400") References: <200707032309.29935.snakeru@gmail.com> Message-ID: <87d4yw4mw3.fsf@mulj.homelinux.net> Alexey Nezhdanov writes: > The most essential for me is freeing all sorts of RAM that python allocated > for objects or whatever else. Reinitializing Python should work, but it might not produce the effect you expect. Python allocates memory using the underlying C functions such as malloc. Shutting down the interpreter simply makes Python call free on the memory segments that belong to Python objects. Because of the way malloc and free work, doing that is no guarantee that memory will be returned to the operating system. free() only guarantees that the memory will be marked as usable for unspecified later calls to malloc. Some malloc implementations go one step further and actually return the memory to the system, but that is only possible when freeing the block(s) at the very end of the region, so you cannot rely on it. At the end of the day, your process will probably not decrease in memory footprint; however, it will not grow on further allocations because it will reuse the freed memory. But, barring a bug in Python, that is exactly the situation you would be in if you didn't reinitialize the interpreter in the first place. > IOW - I want python to be returned in virgin state as it was on my > program load w/o exiting my program and closing all files/tcp > connections that it have opened/or resetting any of other my C data > structures that were created. Doesn't Python automatically close open files when the script finishes? From hniksic at xemacs.org Fri Jul 13 13:27:59 2007 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Fri, 13 Jul 2007 13:27:59 +0200 Subject: [capi-sig] Getting Numbers in C Message-ID: <873azs4k74.fsf@mulj.homelinux.net> > double PyNumber_AsDouble(PyObject * value) > { > if (PyFloat_Check(value) || PyInt_Check(value)) { > return PyFloat_AsDouble(value); > } else { > PyObject *pyfloat; > double d; > > pyfloat = PyNumber_Float(value); > d = PyNumber_AsDouble(pyfloat); I assume this should be PyFloat_AsDouble? > Py_XDECREF(pyfloat); > return d; > } > } What happens if PyNumber_Float raises an exception? As far as I can tell, PyNumber_AsDouble will result in a "bad argument" exception being raised and will return -1, losing track of the original exception in the process. Come to think of it, error checking problems could very well be the reason why the API doesn't provide a PyNumber_AsDouble. From snakeru at gmail.com Fri Jul 13 13:45:43 2007 From: snakeru at gmail.com (Alexey Nezhdanov) Date: Fri, 13 Jul 2007 15:45:43 +0400 Subject: [capi-sig] is deinitialisation of embedded python equal to fullrestart? In-Reply-To: <87d4yw4mw3.fsf@mulj.homelinux.net> References: <200707032309.29935.snakeru@gmail.com> <87d4yw4mw3.fsf@mulj.homelinux.net> Message-ID: <200707131545.43448.snakeru@gmail.com> On Friday 13 July 2007 14:29, Hrvoje Niksic wrote: > Alexey Nezhdanov writes: > > The most essential for me is freeing all sorts of RAM that python > > allocated for objects or whatever else. > > Reinitializing Python should work, but it might not produce the effect > you expect. > > Python allocates memory using the underlying C functions such as > malloc. Shutting down the interpreter simply makes Python call free > on the memory segments that belong to Python objects. Because of the > way malloc and free work, doing that is no guarantee that memory will > be returned to the operating system. free() only guarantees that the > memory will be marked as usable for unspecified later calls to malloc. > Some malloc implementations go one step further and actually return > the memory to the system, but that is only possible when freeing the > block(s) at the very end of the region, so you cannot rely on it. > > At the end of the day, your process will probably not decrease in > memory footprint; however, it will not grow on further allocations > because it will reuse the freed memory. But, barring a bug in Python, > that is exactly the situation you would be in if you didn't > reinitialize the interpreter in the first place. ok, thanks. Still it may be ok for my purposes if I can keep python from growing forever and just keep it beneath some limit. Also, as I understand - memory that is marked as available for reuse will be swapped out if need for RAM will arise. > > IOW - I want python to be returned in virgin state as it was on my > > program load w/o exiting my program and closing all files/tcp > > connections that it have opened/or resetting any of other my C data > > structures that were created. > > Doesn't Python automatically close open files when the script > finishes? Yes, it does. But I meant files that are opened by C part of the program. -- Respectfully Alexey Nezhdanov From cbarton at metavr.com Fri Jul 13 16:08:39 2007 From: cbarton at metavr.com (Campbell Barton) Date: Sat, 14 Jul 2007 00:08:39 +1000 Subject: [capi-sig] Getting Numbers in C In-Reply-To: <873azs4k74.fsf@mulj.homelinux.net> References: <873azs4k74.fsf@mulj.homelinux.net> Message-ID: <46978767.2030302@metavr.com> Hrvoje Niksic wrote: >> double PyNumber_AsDouble(PyObject * value) >> { >> if (PyFloat_Check(value) || PyInt_Check(value)) { >> return PyFloat_AsDouble(value); >> } else { >> PyObject *pyfloat; >> double d; >> >> pyfloat = PyNumber_Float(value); >> d = PyNumber_AsDouble(pyfloat); > > I assume this should be PyFloat_AsDouble? > >> Py_XDECREF(pyfloat); >> return d; >> } >> } > > What happens if PyNumber_Float raises an exception? As far as I can > tell, PyNumber_AsDouble will result in a "bad argument" exception > being raised and will return -1, losing track of the original > exception in the process. > > Come to think of it, error checking problems could very well be the > reason why the API doesn't provide a PyNumber_AsDouble. It is a problem but I dont think thats a good enough reason not to have some way to do this since its such a common operation. PyFloat_AsDouble cant return an error value, so enforcing that the pyobject is checked with PyNumber_Check seems reasonable to me. -- Campbell J Barton (ideasman42) From hniksic at xemacs.org Sun Jul 15 10:45:39 2007 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Sun, 15 Jul 2007 10:45:39 +0200 Subject: [capi-sig] is deinitialisation of embedded python equal tofullrestart? In-Reply-To: <200707131545.43448.snakeru@gmail.com> (Alexey Nezhdanov'smessage of "Fri, 13 Jul 2007 15:45:43 +0400") References: <200707032309.29935.snakeru@gmail.com><87d4yw4mw3.fsf@mulj.homel inux.net><200707131545.43448.snakeru@gmail.com> Message-ID: <87hco6kqbw.fsf@mulj.homelinux.net> Alexey Nezhdanov writes: >> But, barring a bug in Python, that is exactly the situation you >> would be in if you didn't reinitialize the interpreter in the first >> place. > ok, thanks. Still it may be ok for my purposes if I can keep python from > growing forever and just keep it beneath some limit. But Python should behave like that anyway. If your Python is "growing forever", it means that there is a bug in Python, or in the code using it incorrectly. > Also, as I understand - memory that is marked as available for reuse > will be swapped out if need for RAM will arise. Theoretically. In practice, if the memory is interspersed with the chunks of memory that are in use, you get mixed results. From jeff at taupro.com Mon Jul 16 00:58:38 2007 From: jeff at taupro.com (Jeff Rush) Date: Sun, 15 Jul 2007 17:58:38 -0500 Subject: [capi-sig] is deinitialisation of embedded python equal tofullrestart? In-Reply-To: <87hco6kqbw.fsf@mulj.homelinux.net> References: <200707032309.29935.snakeru@gmail.com><87d4yw4mw3.fsf@mulj.homel inux.net><200707131545.43448.snakeru@gmail.com> <87hco6kqbw.fsf@mulj.homelinux.net> Message-ID: <469AA69E.7050806@taupro.com> Hrvoje Niksic wrote: > Alexey Nezhdanov writes: > >>> But, barring a bug in Python, that is exactly the situation you >>> would be in if you didn't reinitialize the interpreter in the first >>> place. >> ok, thanks. Still it may be ok for my purposes if I can keep python from >> growing forever and just keep it beneath some limit. > > But Python should behave like that anyway. If your Python is "growing > forever", it means that there is a bug in Python, or in the code using > it incorrectly. A different but somewhat related problem with Python is that the underlying C memory manager does not release memory back to the operating system when Python releases it to the memory manager. This leads to a situation where you have a momentary need for more memory, say at startup, and then you drop back down in usage but never give that memory to other apps. This is a known problem with the use of Python in embedded devices such as the One-Laptop-Per-Child and the OpenMoko cell phone. I've heard rumors that someone is looking at it, especially because of the OLPC project but have no other information. The Parrot/Pynie project (Python on top of the Perl6 engine) is using memory pools and mark/sweep so they can rearrange objects behind the scenes and release pools back to the OS. Not that that helps us cPython users of course, but I thought it interesting. -Jeff From mal at egenix.com Mon Jul 16 09:58:43 2007 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 16 Jul 2007 09:58:43 +0200 Subject: [capi-sig] is deinitialisation of embedded python equal tofullrestart? In-Reply-To: <469AA69E.7050806@taupro.com> References: <200707032309.29935.snakeru@gmail.com><87d4yw4mw3.fsf@mulj.homel inux.net><200707131545.43448.snakeru@gmail.com> <87hco6kqbw.fsf@mulj.homelinux.net> <469AA69E.7050806@taupro.com> Message-ID: <469B2533.6050405@egenix.com> On 2007-07-16 00:58, Jeff Rush wrote: > Hrvoje Niksic wrote: >> Alexey Nezhdanov writes: >> >>>> But, barring a bug in Python, that is exactly the situation you >>>> would be in if you didn't reinitialize the interpreter in the first >>>> place. >>> ok, thanks. Still it may be ok for my purposes if I can keep python from >>> growing forever and just keep it beneath some limit. >> But Python should behave like that anyway. If your Python is "growing >> forever", it means that there is a bug in Python, or in the code using >> it incorrectly. > > A different but somewhat related problem with Python is that the underlying C > memory manager does not release memory back to the operating system when > Python releases it to the memory manager. This leads to a situation where you > have a momentary need for more memory, say at startup, and then you drop back > down in usage but never give that memory to other apps. > > This is a known problem with the use of Python in embedded devices such as the > One-Laptop-Per-Child and the OpenMoko cell phone. I've heard rumors that > someone is looking at it, especially because of the OLPC project but have no > other information. > > The Parrot/Pynie project (Python on top of the Perl6 engine) is using memory > pools and mark/sweep so they can rearrange objects behind the scenes and > release pools back to the OS. Not that that helps us cPython users of course, > but I thought it interesting. Python 2.5 does the same. Tim Peters (AFAIR) applied some changes to pymalloc to make this possible. Note that you can tune pymalloc using various parameters you can tweak (see obmalloc.c) and even switch it off if you need the OS to be in complete control. Doing so will slow the interpreter considerably. Also note that quite a few object implementations use free lists which are not easy to tweak in this way. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 16 2007) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From cbarton at metavr.com Fri Jul 20 23:30:16 2007 From: cbarton at metavr.com (Campbell Barton) Date: Sat, 21 Jul 2007 07:30:16 +1000 Subject: [capi-sig] Getting Numbers in C In-Reply-To: <46978767.2030302@metavr.com> References: <873azs4k74.fsf@mulj.homelinux.net> <46978767.2030302@metavr.com> Message-ID: <46A12968.5060301@metavr.com> Campbell Barton wrote: > Hrvoje Niksic wrote: >>> double PyNumber_AsDouble(PyObject * value) >>> { >>> if (PyFloat_Check(value) || PyInt_Check(value)) { >>> return PyFloat_AsDouble(value); >>> } else { >>> PyObject *pyfloat; >>> double d; >>> >>> pyfloat = PyNumber_Float(value); >>> d = PyNumber_AsDouble(pyfloat); >> I assume this should be PyFloat_AsDouble? >> >>> Py_XDECREF(pyfloat); >>> return d; >>> } >>> } >> What happens if PyNumber_Float raises an exception? As far as I can >> tell, PyNumber_AsDouble will result in a "bad argument" exception >> being raised and will return -1, losing track of the original >> exception in the process. >> >> Come to think of it, error checking problems could very well be the >> reason why the API doesn't provide a PyNumber_AsDouble. > > It is a problem but I dont think thats a good enough reason not to have > some way to do this since its such a common operation. > > PyFloat_AsDouble cant return an error value, so enforcing that the > pyobject is checked with PyNumber_Check seems reasonable to me. To follow up on this issue, Iv added the function below to a branch of the Blender/Py API so Numbers that aren't pythons Ints of Floats work. ----------------------------------------------------- double EXPP_PyNumber_AsDouble(PyObject * value) { if (PyFloat_Check(value) || PyInt_Check(value)) { return PyFloat_AS_DOUBLE(value); } else { PyObject *pyfloat; double d; pyfloat = PyNumber_Float(value); d = PyFloat_AsDouble(pyfloat); Py_XDECREF(pyfloat); return d; /* will be -1 and set an error if its PyNumber_Float failed */ } } ----------------------------------------------------- from looking at pythons source there are 2 ways to make sure this function succeeds. I was under the impression the only way to do error checking for this was to do.... float f; if (!PyFloat_Check(pyob)) { PyErr_SetString(PyExc_TypeError, "float argument required"); return -1; } f = PyFloat_AS_DOUBLE(pyob) But another way to do it is.... (used in python 2.5.1's stringobject.c:4150) float f = PyFloat_Check(pyob); if (f == -1.0 && PyErr_Occurred()) { PyErr_SetString(PyExc_TypeError, "float argument required"); return -1; } Which would work with EXPP_PyNumber_AsDouble to do error checking as well. so I think error checking is ok with that function. (and would like to see PyNumber_AsDouble in the C/Python API hint, hint) However Im wondering if this is advisable, with PyErr_Occurred() its possible that some other part of the API (even a totaly separate API) sets the error string but dosnt return an error value. - Any suggestions? From hniksic at xemacs.org Sat Jul 21 01:07:57 2007 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Sat, 21 Jul 2007 01:07:57 +0200 Subject: [capi-sig] Getting Numbers in C In-Reply-To: <46A12968.5060301@metavr.com> (Campbell Barton's message of"Sat, 21 Jul 2007 07:30:16 +1000") References: <873azs4k74.fsf@mulj.homelinux.net> <46978767.2030302@metavr.com><46A12968.5060301@metavr.com> Message-ID: <87fy3id67m.fsf@mulj.homelinux.net> Campbell Barton writes: > pyfloat = PyNumber_Float(value); It might be a good idea to return -1 immediately if this returns NULL. That way you prevent clobbering the real exception with an "invalid argument" exception returned by PyFloat_AsDouble. If PyNumber_Float succeeds, but pyfloat turns out not to be float, then the invalid argument exception is perfectly fine. > However Im wondering if this is advisable, with PyErr_Occurred() its > possible that some other part of the API (even a totaly separate > API) sets the error string but dosnt return an error value. - Any > suggestions? Requiring the caller to check for PyErr_Occurred seems like bad style, but I'm not sure that it's an actual problem, as long as it's clearly documented. As far as I know, no part of the API is allowed to set the error string and not return an error value because that breaks other parts of Python. Consider this function: static PyObject * test() { PyErr_SetString(PyExc_TypeError, "test"); Py_INCREF(Py_None); return Py_None; } Calling it subtly breaks Python: >>> import foo >>> foo.test() >>> import sys Traceback (most recent call last): File "", line 1, in TypeError: test The exception lingered until the first function that relies on PyErr_Occurred() to test for an exception, in this case PyImport_*. From cbarton at metavr.com Sat Jul 21 04:33:35 2007 From: cbarton at metavr.com (Campbell Barton) Date: Sat, 21 Jul 2007 12:33:35 +1000 Subject: [capi-sig] Getting Numbers in C In-Reply-To: <87fy3id67m.fsf@mulj.homelinux.net> References: <873azs4k74.fsf@mulj.homelinux.net> <46978767.2030302@metavr.com><46A12968.5060301@metavr.com> <87fy3id67m.fsf@mulj.homelinux.net> Message-ID: <46A1707F.80709@metavr.com> Hrvoje Niksic wrote: > Campbell Barton writes: > >> pyfloat = PyNumber_Float(value); > > It might be a good idea to return -1 immediately if this returns > NULL. That way you prevent clobbering the real exception with an > "invalid argument" exception returned by PyFloat_AsDouble. I think your right, whats there should work but it could return -1 early. > If PyNumber_Float succeeds, but pyfloat turns out not to be float, > then the invalid argument exception is perfectly fine. Is this possible? - if its a number type I would have thaught it must be able to be converted to a float. >> However Im wondering if this is advisable, with PyErr_Occurred() its >> possible that some other part of the API (even a totaly separate >> API) sets the error string but dosnt return an error value. - Any >> suggestions? > > Requiring the caller to check for PyErr_Occurred seems like bad style, > but I'm not sure that it's an actual problem, as long as it's clearly > documented. Lots of Pythons internal code and modules do this. see hows PyFloat_AsDouble's used in cfield.c, _struct.c, datetimemodule.c, selectmodule.c, socketmodule.c..... and others but look in those to see that its used. It looked like bad coding style to me too. > As far as I know, no part of the API is allowed to set the error > string and not return an error value because that breaks other parts > of Python. Consider this function: > > static PyObject * > test() > { > PyErr_SetString(PyExc_TypeError, "test"); > Py_INCREF(Py_None); > return Py_None; > } > > Calling it subtly breaks Python: > >>>> import foo >>>> foo.test() >>>> import sys > Traceback (most recent call last): > File "", line 1, in > TypeError: test > > The exception lingered until the first function that relies on > PyErr_Occurred() to test for an exception, in this case PyImport_*. your example will mess things up but the way the API works (it seems), is you have functions to work around this and return the correct error values with functions that interface python. PyFloat_AsDouble - assumes the type is a PyFloat, if its not compatible it WILL set the error string and leaves the API to see if the error string is set and return an error value (-1 or NULL depending on the type of function). From cbarton at metavr.com Sat Jul 21 05:06:16 2007 From: cbarton at metavr.com (Campbell Barton) Date: Sat, 21 Jul 2007 13:06:16 +1000 Subject: [capi-sig] Getting Numbers in C In-Reply-To: <46A1707F.80709@metavr.com> References: <873azs4k74.fsf@mulj.homelinux.net> <46978767.2030302@metavr.com><46A12968.5060301@metavr.com> <87fy3id67m.fsf@mulj.homelinux.net> <46A1707F.80709@metavr.com> Message-ID: <46A17828.6030201@metavr.com> After all that! I should have looked at the source earlier but BOTH PyFloat_AsDouble and PyNumber_Float support the PyNumber protocol, the ONLY difference is PyNumber_Float will convert a string "0.01". It would be good if sombody on this list could update API docs, Iv mailed some small doc additions to the suggested address for doc updates but no reply. http://www.python.org/doc/api/floatObjects.html The following additions could be made to... double PyFloat_AsDouble( PyObject *pyfloat) This function will also work with a PyInt or any type that implements the PyNumber protocol's (tp_as_number->nb_float) function. double PyFloat_AS_DOUBLE( PyObject *pyfloat) This function will only work with PyFloat or a subtype of PyFloat. From hniksic at xemacs.org Sat Jul 21 09:59:48 2007 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Sat, 21 Jul 2007 09:59:48 +0200 Subject: [capi-sig] Getting Numbers in C In-Reply-To: <46A1707F.80709@metavr.com> (Campbell Barton's message of "Sat,21 Jul 2007 12:33:35 +1000") References: <873azs4k74.fsf@mulj.homelinux.net> <46978767.2030302@metavr.com><46A12968.5060301@metavr.com> <87fy3id67m.fsf@mulj.homelinux.net><46A1707F.80709@metavr.com> Message-ID: <87y7hab30r.fsf@mulj.homelinux.net> Campbell Barton writes: >> If PyNumber_Float succeeds, but pyfloat turns out not to be float, >> then the invalid argument exception is perfectly fine. > > Is this possible? - if its a number type I would have thaught it > must be able to be converted to a float. You're right, PyNumber_Float type-checks the return value of tp_as_number->nb_float. I thought I had checked previously and had seen otherwise, but I must have confused it with something else. From python_capi at behnel.de Sat Jul 28 13:57:41 2007 From: python_capi at behnel.de (Stefan Behnel) Date: Sat, 28 Jul 2007 13:57:41 +0200 Subject: [capi-sig] [ANN] The Cython project Message-ID: <46AB2F35.3050601@behnel.de> Hi all, I just wanted to announce the availability of the first official Cython release (0.9) by the SAGE maintainers William Stein and Robert Bradshaw (and a bit by myself). http://www.cython.org/ Cython is based on the well-known Pyrex translator by Greg Ewing, but supports more cutting edge functionality and optimizations. It was formerly known as the SageX fork of Pyrex. Some of the highlights compared to the latest Pyrex version: 1. Conditional expressions (a if blah else b) 2. List comprehensions 3. Optimized looping (for whatever in blah: is much faster in Cython) 4. Better support for module imports: import a.b.cython_module 5. 64-bit Python 2.5 support and there's a lot more in the queue ... Cython has a site up on launchpad, so if you have any questions or suggestions, found any bugs or want to contribute patches, please visit the project there: https://launchpad.net/cython/ Have fun, Stefan From cbarton at metavr.com Sun Jul 29 04:16:47 2007 From: cbarton at metavr.com (Campbell Barton) Date: Sun, 29 Jul 2007 12:16:47 +1000 Subject: [capi-sig] C/API error handling Message-ID: <46ABF88F.5020206@metavr.com> In the Python C api is that your supposed to check every operation succeeds. - Append, a new list, a new float etc. This makes sense, since somebody could append to a list until the system is out of memory.. or whatever. The thing that bothers me is theres no good way to test the error cases. For example in Blender3D and PyGame (the only 2 C API's I'v looked at) - many checks for failed operations are missing. so its possible somebody could run out of ram and crash the API with python. for Blender3D or PyGame it probably dosnt matter a great deal, but in other cases, you'd want to make sure that doing crazy stuff within an exception (for example) wont crash the application. Not just to look at the code and think it should work but actually run the error case. Is there any way to do this? I was thinking there could be a debug mode where python/C API functions like PyObject_New, PyList_Append etc randomly fail a percentage of the time. Then you could run unit tests in a debugger and any crashes could be traced. This isnt an exact approach but if the unit tests run enough times you could be fairly sure its well tested (assuming the unit tests cover the API). Anyone tried this before or does something like this already exist? -- Campbell J Barton (ideasman42) From cbarton at metavr.com Sun Jul 29 11:44:16 2007 From: cbarton at metavr.com (Campbell Barton) Date: Sun, 29 Jul 2007 19:44:16 +1000 Subject: [capi-sig] C/API error handling In-Reply-To: <46AC3690.5080608@gmail.com> References: <46ABF88F.5020206@metavr.com> <46AC3690.5080608@gmail.com> Message-ID: <46AC6170.1030403@metavr.com> Hi Joe, was thinking of messing with the memory allocator yes. maybe there are other areas also, but the memory allocator would be a good place to start. This could be a compile time option for python, maybe a module thats only built in debug mode, or a seperate option. --- import CAPIForceError CAPIForceError.setmemfail(0.01) ...test the apis error stability... --- So python could initialize etc without failing, and the test code would have to set the error probability before it runs a script that tests the api. Joe Eagar wrote: > One thing you could do is hack the python allocator to fail every once > in a while. However, this is likely to cause problems within python > itself, along with problems with a python api codebase. > > The allocator is in Objects/obmalloc.c in the python source, iirc. > > Joe > > Campbell Barton wrote: >> In the Python C api is that your supposed to check every operation >> succeeds. - Append, a new list, a new float etc. >> >> This makes sense, since somebody could append to a list until the >> system is out of memory.. or whatever. >> >> The thing that bothers me is theres no good way to test the error cases. >> >> For example in Blender3D and PyGame (the only 2 C API's I'v looked at) >> - many checks for failed operations are missing. so its possible >> somebody could run out of ram and crash the API with python. >> >> for Blender3D or PyGame it probably dosnt matter a great deal, but in >> other cases, you'd want to make sure that doing crazy stuff within an >> exception (for example) wont crash the application. >> Not just to look at the code and think it should work but actually run >> the error case. >> >> Is there any way to do this? >> >> I was thinking there could be a debug mode where python/C API >> functions like PyObject_New, PyList_Append etc randomly fail a >> percentage of the time. >> >> Then you could run unit tests in a debugger and any crashes could be >> traced. >> This isnt an exact approach but if the unit tests run enough times you >> could be fairly sure its well tested (assuming the unit tests cover >> the API). >> >> Anyone tried this before or does something like this already exist? >> >> > > -- Campbell J Barton (ideasman42)