From jon at csh.rit.edu Thu Jan 3 08:18:08 2008 From: jon at csh.rit.edu (Jon Parise) Date: Thu, 3 Jan 2008 02:18:08 -0500 Subject: [capi-sig] Deploying embedded Python In-Reply-To: <476B6B9B.9090907@taupro.com> References: <47670F54.20104@gmx.de> <476B6B9B.9090907@taupro.com> Message-ID: <20080103071808.GA12009@csh.rit.edu> On Fri, Dec 21, 2007 at 01:30:35AM -0600, Jeff Rush wrote: > > 2) How to isolate the embedded interpreter from environmental effects. I > > have found that on occasion, the interpreter would pick up "stray" > > installations which can cause weird problems. Which environmental > > settings affect the startup of an embedded Python interpreter? How does > > one work around/remove those dependencies? Is there any information > > available about how exactly the startup works? What is being read/loaded > > in which order etc? > > If you are picking up stray installations it is probably through the > PYTHONPATH environment variable. For a brief understanding of these, run the > command "python --help". For your embedded world, you ought to change your > distributed code to use a different environment variable, maybe a prefix or > suffix. You can also set the Py_IgnoreEnvironmentFlag to 0 before calling Py_Initialize(). This is the equivalent of running python with the -E command line option: -E : ignore environment variables (such as PYTHONPATH) -- Jon Parise (jon of csh.rit.edu) :: "Scientia potentia est" From candyshop999 at gmail.com Fri Jan 4 13:22:34 2008 From: candyshop999 at gmail.com (Super Star) Date: Fri, 4 Jan 2008 14:22:34 +0200 Subject: [capi-sig] Palm Vacations, A Travel Guide For Cologne Message-ID: Palm Vacations, A Travel Guide For Cologne Cologne is the four most populated cities in Germany. These days the city is a beautiful cultured destination with great leisure facilities, pubs and shops. However, during WWII, Cologne was heavily bombed and damaged. After years of restoration work it has been rebuilt to portray the wonderful historic Roman city it use to be. Travelling to Cologne Flying is the recommended choice of transport as the city has K?ln/Bonn Konrad Adenauer Airport located 10km from the centre. Also, for people arriving from countries that are outside Europe, Frankfurt Airport is the closest and provides transfers for the 200km journey. Trains and large roads make it easy to get to the city via other methods. Accommodation in Cologne Hotels are the most popular choice of accommodation in Cologne. However, check for the festival dates as this can seriously affect the availability of rooms. Hotels are great as they offer different classes of rooms and are conveniently situated around the city. A Brief History of Cologne Cologne has a vast history as it is the eldest city in Germany. The city has great importance in Roman History as it is one of four sanctified cities that formed the north corner spot of the Empire. At the beginning of the thirteenth century Cologne was renowned for being the largest fortified city in the world. However, it has been changed around a lot since then as 90% of Cologne was destroyed during WWII. City Attractions and Sights Since the heavy bombing in WWII, restoration efforts have seen the city burgeon into the historic Roman centre that it used to be. The Cologne Cathedral (K?lner Dom) that was originally established in 1248 was rebuilt to its previous state. It is an important feature of the city and one of the worlds most impressive Cathedrals. If you are interested in museums then head for the Aldstadt city?s most famous museums are such as the Museum Ludwig. For relaxation check out Claudius Spa that is found in the Rheinpark area although, you will need a car to make the journey. Shopping in Cologne Cologne has a great range of shops that cater for everybody?s needs. The most popular shopping areas are in the centre where there are a number of large. Other smaller streets that head out of the centre have more unique shops and indoor markets. The shopping areas are sectioned off for pedestrians, attractive and well laid out. Food and Drink Cologne has a great range of top quality restaurants that offer all kinds of international food. There are also numerous attractive cafes and bars as well as all takeaway options. From tepperly at llnl.gov Fri Jan 11 19:14:36 2008 From: tepperly at llnl.gov (Tom Epperly) Date: Fri, 11 Jan 2008 10:14:36 -0800 Subject: [capi-sig] Question about proper GIL management for a language interoperability tool Message-ID: <4787B20C.6040404@llnl.gov> I working on a language interoperability tool called Babel, https://computation.llnl.gov/casc/components/. It provides high-performance, bi-directional communication between any combination of C, C++, F77, F90, Java, and Python. Right now, I am trying to make sure I am handling the GIL locking and unlocking correctly, and I am wondering if the Python community can confirm whether I am handling things appropriately. Babel handles the many-to-many language interoperability problem with a hub and spoke architecture, so all inter-language calls go through C. Hence, I can discuss how I am handling the GIL in terms of communications between C and Python. I am going to describe what I am doing using pseudo-code and leaving out irrelevant details. CASE 1 (some other language calling Python via C) ======================================== foo_Factorial(struct foo_t *self, int32_t arg) { /* C routine that calls Python */ PyGILState_STATE _gstate; _gstate = PyGILState_Ensure(); /* convert incoming arguments from C to Python */ _result = PyObject_CallObject(_pfunc, _args); /* process outgoing arguments or exceptions */ PyGILState_Release(_gstate); /* return to caller in whatever language */ } CASE 2 (Python calling some other language via C) ======================================== static PyObject * pStub_foo_Factorial(PyObject *_self, PyObject *_args, PyObject *_kwdict) { PyObject *result; /* unpack incoming Python arguments into their C equivalents */ Py_BEGIN_ALLOW_THREADS /* dispatch to method implementation */ foo_Factorial(i,k,l); /* this step is actually done through a function pointer */ Py_END_ALLOW_THREADS /* pack outgoing C arguments into the Python return value or process any exceptions */ return result; } Discussion ======================================== Case 1 seems pretty straight forward. The current thread must own the GIL before making any Python C API calls, and it should release it when it's done. Case 2 also seems appropriate because function x_y_Method could very well do blocking I/O operation. For example, Babel also supports remote method invocation (RMI), so x_y_Method might dispatch the method across the network to another machine. In ceval.h where Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS are defined, it says "WARNING: NEVER NEST CALLS TO Py_BEGIN_ALLOW_THREADS AND Py_END_ALLOW_THREADS!!!". This makes me wonder if there is an issue with Python calling itself through Babel. Let's assume that Python is the main driver, and it calls a Babel wrapped Python method to calculate factorial or something. This will result in nested calls to PyGILState_Ensure/PyGILState_Release and Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS. Python interactive shell (invokes foo.Factorial(3)) Py_BEGIN_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) PyGILState_Ensure (in foo_Factorial (case 1) arg=3) Py_BEGIN_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) PyGILState_Ensure (in foo_Factorial (case 1) arg=2) Py_BEGIN_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) PyGILState_Ensure (in foo_Factorial (case 1) arg=1) PyGILState_Release (in foo_Factorial (case 1) arg=1) Py_END_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) PyGILState_Release (in foo_Factorial (case 1) arg=2) Py_END_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) PyGILState_Release (in foo_Factorial (case 1) arg=3) Py_END_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) Python interactive shell Is this approach correct? Regards, Tom Epperly LLNL/CASC From Jack.Jansen at cwi.nl Sat Jan 12 18:28:26 2008 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Sat, 12 Jan 2008 18:28:26 +0100 Subject: [capi-sig] Question about proper GIL management for a language interoperability tool In-Reply-To: <4787B20C.6040404@llnl.gov> References: <4787B20C.6040404@llnl.gov> Message-ID: <8E5DD01D-C64B-4BB9-B834-B85F36DF868B@cwi.nl> On 11-Jan-2008, at 19:14 , Tom Epperly wrote: > CASE 1 (some other language calling Python via C) > ======================================== > foo_Factorial(struct foo_t *self, int32_t arg) { > /* C routine that calls Python */ > PyGILState_STATE _gstate; > _gstate = PyGILState_Ensure(); > /* convert incoming arguments from C to Python */ > _result = PyObject_CallObject(_pfunc, _args); > /* process outgoing arguments or exceptions */ > PyGILState_Release(_gstate); > /* return to caller in whatever language */ > } This is what I always do too. > CASE 2 (Python calling some other language via C) > ======================================== > static PyObject * > pStub_foo_Factorial(PyObject *_self, PyObject *_args, PyObject > *_kwdict) { > PyObject *result; > /* unpack incoming Python arguments into their C equivalents */ > Py_BEGIN_ALLOW_THREADS > /* dispatch to method implementation */ > foo_Factorial(i,k,l); /* this step is actually done through a > function > pointer */ > Py_END_ALLOW_THREADS > /* pack outgoing C arguments into the Python return value or process > any exceptions */ > return result; > } Here I use PyThreadState *_save = PyEval_SaveThread(); ... PyEval_RestoreThread(_save); But that pretty much amounts to the same, I think, modulo C preprocessor trickery (which got in my way because I'm working in C++). This module does serious multithreaded nesting from Python to C++ to Python to C++ etc etc etc, and I haven't yet run into a problem. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From rhamph at gmail.com Sat Jan 12 18:40:05 2008 From: rhamph at gmail.com (Adam Olsen) Date: Sat, 12 Jan 2008 10:40:05 -0700 Subject: [capi-sig] Question about proper GIL management for a language interoperability tool In-Reply-To: <4787B20C.6040404@llnl.gov> References: <4787B20C.6040404@llnl.gov> Message-ID: On Jan 11, 2008 11:14 AM, Tom Epperly wrote: > In ceval.h where Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS are > defined, it says "WARNING: NEVER NEST CALLS TO Py_BEGIN_ALLOW_THREADS > AND Py_END_ALLOW_THREADS!!!". This makes me wonder if there is an issue > with Python calling itself through Babel. Let's assume that Python is > the main driver, and it calls a Babel wrapped Python method to calculate > factorial or something. This will result in nested calls to > PyGILState_Ensure/PyGILState_Release and > Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS. The comment is misleading. It's referring to a literal nesting, in one function: int foo(void) { Py_BEGIN_ALLOW_THREADS PyGILState_Ensure(); Py_BEGIN_ALLOW_THREADS do_something(); if (failed) { Py_BLOCK_THREADS return -1; } Py_END_ALLOW_THREADS PyGILState_Release(); Py_END_ALLOW_THREADS return 0; } Note the Py_BLOCK_THREADS in the middle. It will only handle the inner BEGIN/END pair, so the outer pair will never get undone. Then again, you'll never call PyGILState_Release() either, so this example is inherently hosed. Your approach, where you're calling different functions and they all clean up after themselves, seems just fine. > > Python interactive shell (invokes foo.Factorial(3)) > Py_BEGIN_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) > PyGILState_Ensure (in foo_Factorial (case 1) arg=3) > Py_BEGIN_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) > PyGILState_Ensure (in foo_Factorial (case 1) arg=2) > Py_BEGIN_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) > PyGILState_Ensure (in foo_Factorial (case 1) arg=1) > PyGILState_Release (in foo_Factorial (case 1) arg=1) > Py_END_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) > PyGILState_Release (in foo_Factorial (case 1) arg=2) > Py_END_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) > PyGILState_Release (in foo_Factorial (case 1) arg=3) > Py_END_ALLOW_THREADS (in pStub_foo_Factorial (case 2)) > Python interactive shell -- Adam Olsen, aka Rhamphoryncus From hniksic at xemacs.org Thu Jan 17 13:49:55 2008 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Thu, 17 Jan 2008 13:49:55 +0100 Subject: [capi-sig] Register function to call at shutdown Message-ID: <87zlv4obv0.fsf@mulj.homelinux.net> Is it possible to register a function to be called at finalization time? By finalization time I'm referring to the point at which Python is still operational, but is shutting down, for example the time when the modules are getting cleaned up. Functions registered with Py_AtExit get called too late, when it's no longer allowed to invoke any Python API function. I currently simulate this using an object whose __del__ invokes my function, and sticking that object in a private module. During module cleanup Python unhooks the object from the module, firing its __del__ which calls my function. But I consider it a hack, and a fragile one at that -- if anyone deletes the object from the module, the function will be called too early. I'd prefer a cleaner way to deal with this if possible. From mal at egenix.com Fri Jan 18 10:45:38 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 18 Jan 2008 10:45:38 +0100 Subject: [capi-sig] Register function to call at shutdown In-Reply-To: <87zlv4obv0.fsf@mulj.homelinux.net> References: <87zlv4obv0.fsf@mulj.homelinux.net> Message-ID: <47907542.3030802@egenix.com> On 2008-01-17 13:49, Hrvoje Niksic wrote: > Is it possible to register a function to be called at finalization > time? By finalization time I'm referring to the point at which Python > is still operational, but is shutting down, for example the time when > the modules are getting cleaned up. Functions registered with > Py_AtExit get called too late, when it's no longer allowed to invoke > any Python API function. > > I currently simulate this using an object whose __del__ invokes my > function, and sticking that object in a private module. During module > cleanup Python unhooks the object from the module, firing its __del__ > which calls my function. But I consider it a hack, and a fragile one > at that -- if anyone deletes the object from the module, the function > will be called too early. I'd prefer a cleaner way to deal with this > if possible. Why don't you use the standard atexit module for this ? This still only works from Python, but you can add your function in a much more reliable way. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 18 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From hniksic at xemacs.org Fri Jan 18 12:21:07 2008 From: hniksic at xemacs.org (Hrvoje Niksic) Date: Fri, 18 Jan 2008 12:21:07 +0100 Subject: [capi-sig] Register function to call at shutdown In-Reply-To: <47907542.3030802@egenix.com> (M.'s message of "Fri\, 18 Jan 2008 10\:45\:38 +0100") References: <87zlv4obv0.fsf@mulj.homelinux.net> <47907542.3030802@egenix.com> Message-ID: <874pdbbcrg.fsf@mulj.homelinux.net> "M.-A. Lemburg" writes: > Why don't you use the standard atexit module for this ? That's exactly what I was looking for, thanks! (Don't know how I missed it.) > This still only works from Python, but you can add your function in > a much more reliable way. That's not a problem because I can still access it from Python/C using PyModule_Import and friends.