How to call python from a foreign language thread (C++)

[Philip Semanchuk]

On Feb 3, 2009, at 11:37 PM, Victor Lin wrote:

> It does not work. But however, thanks your help. I have tired so many
> methods to do. But it
> crash...crash..deadlock...deadlock..crash...crash... I have no any
> tried success. I am going crazy. Could someone help me, thanks.


Hi Victor,
I have some code that works, although I'm not terribly confident of  
it. The Python documentation in this area is as clear as mud. I'd be  
interested to see if my code works for you as well.

This is part of my function that the Python code calls to set up the  
callback:

if (!PyEval_ThreadsInitialized()) {
    DPRINTF("calling PyEval_InitThreads()\n");
    PyEval_InitThreads();
    // PyEval_InitThreads() acquires the GIL on my behalf but
    // I don't want it at the moment.
    PyEval_ReleaseLock();
}

This sets up the GIL if necessary (i.e. if this is a single-threaded  
program) and is a no-op otherwise (i.e. if the app has already created  
a Python thread).

Then I have this function to perform the callback. It is invoked in a  
new C thread. Comments are inline.

void process_notification(union sigval notification_data) {
     /* Invoked by the system in a new thread as notification of a  
message
        arriving in the queue. */
     PyObject *arglist;
     PyObject *result;
     PyGILState_STATE gstate;
     PyThreadState *main_thread;
     PyThreadState *callback_thread;
     MessageQueue *self = notification_data.sival_ptr;

     DPRINTF("C thread %ld invoked\n", pthread_self());

     // PyGILState_Ensure() implicitly acquires the GIL so I don't need
     // to call PyEval_AcquireLock().
     DPRINTF("Calling PyGILState_Ensure()\n");
     gstate = PyGILState_Ensure();

     // Get the current thread state so that I have an interpreter to
     // which to point.
     DPRINTF("Calling PyThreadState_Get()\n");
     main_thread = PyThreadState_Get();

     // Create a new Python thread for the callback.
     DPRINTF("Calling PyThreadState_New()\n");
     callback_thread = PyThreadState_New(main_thread->interp);

     // Make the callback thread current.
     DPRINTF("Calling PyThreadState_Swap()\n");
     PyThreadState_Swap(callback_thread);

     // Perform the callback.
     arglist = Py_BuildValue("(O)", self->notification_function_param);
     result = PyEval_CallObject(self->notification_function, arglist);
     Py_DECREF(arglist);

     DPRINTF("Done calling\n");

     // Clean up my internal pointers
     Py_XDECREF(self->notification_function);
     Py_XDECREF(self->notification_function_param);
     self->notification_function = NULL;
     self->notification_function_param = NULL;

     // Now unwind the Python thread/GIL stuff above
     DPRINTF("Calling PyThreadState_Swap()\n");
     PyThreadState_Swap(main_thread);

     DPRINTF("Calling PyThreadState_Clear()\n");
     PyThreadState_Clear(callback_thread);

     DPRINTF("Calling PyThreadState_Delete()\n");
     PyThreadState_Delete(callback_thread);

     // PyGILState_Ensure() acquires the lock, but does  
PyGILState_Release()
     // release it? The documentation doesn't say, but it seems like  
it does.
     DPRINTF("Calling PyGILState_Release()\n");
     PyGILState_Release(gstate);

     DPRINTF("exiting thread\n");
};



This code works (in my limited testing) regardless of whether or not  
the Python code has created a thread. For the threaded test, I created  
a background thread that prints "ding!" every second. That thread  
continued to run even after my callback thread was invoked which I  
assume means that I released the GIL properly.

As I mentioned before, this is part of my posix_ipc extension. This  
code (assuming I feel confident enough to release it) will be in the  
next version that should be out soon, so you will have a full working  
example with which to experiment.

HTH,
Philip