[C++-sig] Deriving Python function from C++ base class with pure virtual function

[John Meinel]

Niall Douglas wrote:
> On 13 Nov 2004 at 1:53, Paul F. Kunz wrote:
> 
> 

[...]

>
>>>   Should I start the second thread with Python's own thread class? Is
>>>there some Python C API functions I could call to read the Python
>>>object to be called from this second thread?   Any suggestions /
>>>advise would be appreciated.
> 
> 
> Before entering python, you must ensure that python's thread state 
> has been set for the current thread. This implies holding python's 
> Global Interpreter Lock. If you try entering python without setting 
> the thread state, you'll get a warning on stderr and a segfault most 
> probably.
> 
> If however you want to have multiple threads working with python 
> concurrently, that is a whole new bag of worms. There is a solution 
> but it involves patching BPL to release and grab the python GIL every 
> time it enters and exits C/C++ code. Do say if you want to do this, 
> but be warned it's not trivial.
> 
> Cheers,
> Niall
> 
> 

I've done some work with multiple threads in python. Here is what I did.

First, I have C++ functions that run for a long time doing analysis, and 
some that are very quick. I don't bother threading the quick ones, just 
the long running ones.

I have a Callback class which handles the reporting of how things are 
going. The C++ functions don't realize that they are being called from 
python, because their only interaction with the outside world is through 
the Callback object. The Callback is a C++ object, which I inherit from 
in python.

All of my threads are started from python, which means that they already 
have thread state associated with them, and when the call is made to the 
C++ code, they already have the GIL (or they wouldn't be running.)

I created a class which when instantiated replaces the Callback object 
with one of it's own, such that whenever a call is made, it grabs the 
GIL, and then places the call to the original Callback, and when that 
returns, it releases the GIL.

So what does this all look like.

The C++ side:

class Callback
{
public:
    int send_message(const std::string &msg) = 0;
};

class Worker
{
    Callback *cb_;
public:

    Callback *setCallback(Callback *cb)
    {
      Callback oldcb = cb_;
      cb_ = cb;
      return oldcb;
    }

    int doSomethingFast(int arg);
    int doSomethingLong(int arg);

};

class PyCallback
{
   Callback *other_;
   PyThreadState *state_;
   Worker *worker_;
public:
   PyCallback ()
     : state_(0), other_(0), worker_(0)
   { }
   ~PyCallback ()
   {
     releaseCallback();
   }

   void grabCallback(Worker& w)
   {
     other_ = w.setCallback(this);
     worker_ = &w;
     state_ = PyEval_SaveThread();
   }

   void releaseCallback()
   {
     if(other_) {
       PyEval_AcquireThread(state_);
       worker_->setCallback(other_);
       worker_ = 0;
       state_ = 0;
       other_ = 0;
     }
   }

   int send_message(const std::string &msg)
   {
     int response = 0;
     if (other_) {
       PyEval_AcquireThread(state_);
       response = other_->send_message(msg);
       PyEval_ReleaseThread(state_);
     }
     return response;
   }

};

And now the boost::python wrapper

static int workerDoSomething(Worker& w, int arg)
{
   PyCallback pcb;
   pcb.getCallback(w);
   return w.doSomething(arg);
}

void exportWorker()
{
   class_<Worker>("Worker")
     .def("doSomethingLong", &workerDoSomething)
     .def("doSomethingFast", &Worker::doSomethingFast)
   ;
}

Now, I happen to have a structure where everything that needs work done 
inherits from a "Worker" class. So they all have the setCallback() 
function, and they all use the Callback to talk about what is going on. 
Mine is also a little bit more complicated, as I have another parent 
class, which lets me write more like: (following RAII rules)

static int workerDoSomething(Worker& w, int arg)
{
   PyWrap pw(w);
   return w.doSomething(arg);
}

Also, I don't use raw pointers, everything is a boost::shared_ptr<>, but 
I figured raw pointers is a little bit easier to read (and less to type).

I could see easily turning PyCallback into a template function, such 
that Worker is not defined, and you just need a function like 
setCallback() to exist.

But this is basically what I am doing. Now probably boost::python could 
be rewritten such that it automatically does most of this stuff. (it 
already is creating function wrappers, it could probably just add some 
more threading code into them.)

But this is a way to do it without changing any of boost::python.

It depends on your framework, though.

John
=:->
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 256 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20041114/6f734119/attachment.pgp>