Py_NewInterpreter(), is this a bug in the python core?

[freesteel]

freesteel wrote:
> Yes, I see that now in the documentation, which to me is quite
> confusing.
> So, how do you use python in a multithreaded environment, where for
> example you want to run some embeded python code from a number of
> different C threads?
>
> This article: http://www.linuxjournal.com/article/3641 is quite good,
> but must have been written before these PyGILState_* functions.
>
> I only used Py_NewInterpreter to have a fresh 'import sys'.
>
> Somebody enlighten me, please.
>
> Martin

If you try to replicate the code from the linux journal
(http://www.linuxjournal.com/article/3641) and compile with Py_DEBUG
defined, when running it you will find that you get an exception and a
fatal error message from the python core. The exception is thrown from
pystate.c, line 306:
 			Py_FatalError("Invalid thread state for this thread");

The exception is thrown in my understanding of the code because there
can only ever be one thread state. If  this is true a function to swap
thread states seems rather pointless.
Now, reading up about how to call C API to python from a C thread I
find that with version 2.3 the function pair
PyGILState_Ensure/PyGILState_Release was introduced. This allows
'grabbing' the global interpreter lock, calling your embedded python
code and release the GIL at the end again.

Am I right in my understanding that the use of this PyGILState_* pair
is meant to 'replace' the prologue of creating a new thread state from
the main thread state, swapping it with the current state, doing your
Python/C API calls and then in an epilogue swap the previous thread
state back in, and deletie the now obsolete previously created thread
state?

At least that how I understand the motivation behind the introduction
of tPyGILState_*, read here:
http://www.python.org/dev/peps/pep-0311/

Now, I tried to use this mechanism, but I am not really successful with
it. Basically, in my C thread, I wrap a number of calls to embedded
python (a few PyRun_SimpleString calls, nothing really fancy) with
PyGILState_Ensure and PyGILState_Release. The first C thread also
initializes python as well as initializes threading:
			Py_Initialize();
			PyEval_InitThreads();

(If it helps I can post my whole code here).

When I test this I get deadlocks in most cases after the first thread
has called PyGILState_Release. All other threads at that point 'freeze'
and never return from whatever Python/C API function they are in. I
don't understand why. Moreover, this behavious is not always
reproducable, sometimes the code works as intended. Must be some kind
of race condition? Who can help?

I read that the problem of calling python from a multithreaded
application per thread is very difficult, but a few projects have
managed to solve it, but now we have these PyGILStates and all is much
easier?

Martin