How to call python from a foreign language thread (C++)

[Philip Semanchuk]

On Feb 3, 2009, at 12:51 PM, Victor Lin wrote:

>
> It seems that my program can't call to Python's function from thread
> directly, because there is Global Interpreter Lock. The python's GIL
> is so complex, I have no idea how it works. I'm sorry, what I can do
> is to ask. My question is. What should I do before and after I call a
> Python function from threads?


Hi Victor,
I asked a similar question yesterday and have gotten no response yet  
-- I hope you'll have better luck. I'm writing a C extension that  
wants to implement a callback in a new thread. Here's what I think  
I've learned from reading the threads API doc. Please note that this  
is a classic case of the blind leading the blind! I'm sure some (most?  
all?) of the ideas below are wrong, but I'm hoping that thinking  
through some of this "out loud" will help both of us. Or maybe some  
more knowledgeable person will take pity on/be appalled by my  
ignorance and come to our rescue. =)

Python's infamous GIL doesn't exist when a program is single-threaded.  
Before a new thread is created, the main thread must call  
PyEval_InitThreads() to create the GIL. However, "It is not safe to  
call this function when it is unknown which thread (if any) currently  
has the global interpreter lock." Therefore my extension must do this:

i_have_the_gil = 0;
if (!PyEval_ThreadsInitialized()) {
    PyEval_InitThreads();
    /* "...when this function initializes the lock, it also acquires  
it." */
    i_have_the_gil = 1;
}

That ensures that the GIL is created.

My extension will be calling from a newly-created C thread. The  
*Python* thread doesn't exist yet; so next I have to create it.  
Therefore --

if (!i_have_the_gil)
    PyEval_AcquireLock();

// Might not actually be the main thread but conceptually
// it is OK to assume so here.
main_thread = PyThreadState_Get();

callback_thread = PyThreadState_New(main_thread->interp);

PyThreadState_Swap(callback_thread);

gstate = PyGILState_Ensure();

call_callback_function();

// Now unwind the above

PyGILState_Release(gstate);

PyThreadState_Swap(main_thread);

PyThreadState_Clear(callback_thread);

PyThreadState_Delete(callback_thread);

PyEval_ReleaseLock();


I haven't yet tested this! But hopefully it is the right idea and just  
needs a little fine tuning.

I found this discussion useful:
http://mail.python.org/pipermail/python-list/2006-November/413088.html

It includes the quote, "The current thread state API doc, as you read  
it from top to bottom now, is in fact totally confusing for anyone who  
didn't develop Python himself"

I like! =)

Cheers
Philip