[C++-sig] boost-python callbacks and GIL lock

[David A Wagner]

I'm building an interface between python and a middleware system (DDS) 
with a C++ API where I want to
be able to use event threads in the middleware framework to drive 
callback events in python.  I.e., the python
app can register a callback handler object with the API to be notified 
when messages arrive on a middleware
topic.  Using boost-python I was able to get this to work with only a 
few hitches, which I'd like to ask about.

First, let me say that boost-python was a HUGE time saver here. Thanks a 
ton to those who developed it!

Basically, I defined a callback interface class in C++ with a virtual 
callback method:

class UpdateHandler { virtual void update(Message&); }

where Message is my C++ message data class.  I used the boost-python 
wrappers to export these classes
to Python.  E.g.,:

   class_<UpdateHandlerWrapper, boost::noncopyable>("UpdateHandler")
                .def("update", pure_virtual(&UpdateHandler::update))
                ;

with the special callback wrapper function

    struct UpdateHandlerWrapper : UpdateHandler, 
boost::python::wrapper<UpdateHandler> {
        void update(const Message& val) {
                // Call the virtual function in 
python                         
                this->get_override("update")(val);
     }}

This all worked great as long as  I was using test code that  triggered 
the callback  from within python.
When we tried to use the actual external threads to pump the events, the 
program crashed.

I didn't see anything about this in the boost-python documentation, but 
I think the problem had to do with
thread re-entrance in python.  Specifically, since these event threads 
originate outside of python and its thread
management context, extra code is needed in these callback wrappers to 
acquire python's Global Interpreter
Lock (GIL) in order to enter python's interpreter context in a 
thread-safe manner:

               PyGILState_STATE gstate = PyGILState_Ensure();
                // Call the virtual function in 
python                         
                this->get_override("update")(val);
                PyGILState_Release(gstate);

That seemed to do the trick, but I'd like to get some verification that 
this was the appropriate thing to do
here.  Is this right?  It also makes sense that a call originating from 
python would already have this
context, but some foreign thread would not.

Thanks