Need some help with Python/C api and threading

[Steve Menard]

Les Smithson wrote:
>>>>>>"Steve" == Steve Menard <steve.menard at videotron.ca> writes:

<SNIP>

> 
> I haven't done this for a while and I'm a little hazy on it, so this
> may be incorrect:
> 
> I used 'PyThreadState *ts = Py_NewInterpreter();' to set a new
> sub-interpreter state if called in a new thread.
> 
> If the embedded script calls back into the extension, it restores that
> thread state and acquires the GIL before making any other Py* calls by
> calling 'PyEval_RestoreThread(ts);'. Before returning, it calls
> 'PyEval_SaveThread()'.
> 
> 

Thanks, however I dont think thid will work. The doc for 
Py_NewInterpreter says that it created a "an (almost) totally separate 
environment for the execution of Python code. In particular, the new 
interpreter has separate, independent versions of all imported modules". 
This is not good for me, as the callbacks must come in the "main" 
interpreter context.

Is there a tutorial somewhere? Or a particularly well written extension 
module whose source code I could take a look at?

Let me summarize my situation :

I am writing a python extension, not embedding python. As such, I have 
no control over the interpreter, or the threads.

The library I am embedding is not of my own writing. It can create any 
number of threads. It can make callbacks into the Python interpreter on 
any such thread.

A given thread can original either in python or the library, but control 
can go back and forth : A python method can call a library method, which 
in turn calls back into python, which calls a linrary method, etc ... 
This is a potential problem, because trying to grab in GIL twice from 
the same thread will cause a deadlock.


So far, here is what I am doing (without success).

1) In the init_XXX method, I call PyEval_InitThreads().

2) Every time I pass control to the library, I wrap the call into a 
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS pair. Note that I adde this 
recently, and get an error the second time Py_BEGIN_ALLOW_THREADS is 
called, with the following error "Fatal Python error: PyEval_SaveThread: 
NULL tstate"

3) Finally, whenever I receive a callback from the library, I added 
these lines to the start and end of the method :

		PyInterpreterState* interp = PyInterpreterState_New();
		PyThreadState *tstate = PyThreadState_New(interp);
		PyEval_AcquireThread(tstate);

and

		PyEval_ReleaseThread(tstate);
		PyThreadState_Delete(tstate);
		PyInterpreterState_Delete(interp);


Thats about it. I am sure someone, somewhere has done what I need :(


Thanks for any help you can provide,

Steve