PyThreadState_Swap difference in 2.3.2?

[Mark Hammond]

vincent wehren wrote:
> "Paul Miller" <paul at fxtech.com> schrieb im Newsbeitrag
> news:knbstvgsn3o3qmmtu975g8eb94rhpmae2o at 4ax.com...
> | I have a C++ application that uses multiple embedded Python interpreters.
> I
> | allocate multiple interpreters using Py_NewInterpreter, and switch between
> | them using PyThreadSate_Swap.
> |
> | In 2.2.2, this all worked fine.
> |
> | I just installed 2.3.2, but now the first time I call Py_NewInterpreter,
> it
> | bails out in PyThreadState_Swap, in the debug checking code, with:
> |
> | Py_FatalError("Invalid thread state for this thread");
> |
> | Has the interpreter/thread setup code changed since 2.2?
> |
> | To be complete, I set up with this:
> |
> | Py_Initialize();
> | PyThreadState *globalState = PyThreadState_Get();
> |
> | // allocate new interpreter
> | PyThreadState *interp = Py_NewInterpreter();
> 
> The same thing stumped me too a couple of days ago (s.
> http://groups.google.nl/groups?q=PyNew_Interpreter&hl=nl&lr=&ie=UTF-8&oe=UTF-8&selm=braiu1%243r3%241%40news4.tilbu1.nb.home.nl&rnum=1 )
> with 2.3.3c1 (same applies 2.4a0 for that matter).
> 
> Although I  am sure the following snip in PyThreadState_Swap was put there
> for a reason,
> it would be great if someone could put some light on the matter....:
> 
>  /* It should not be possible for more than one thread state
>     to be used for a thread.  Check this the best we can in debug
>     builds.
>  */
> #if defined(Py_DEBUG) && defined(WITH_THREAD)
>  if (new) {
>   PyThreadState *check = PyGILState_GetThisThreadState();
>   if (check && check != new)
>    Py_FatalError("Invalid thread state for this thread");
>  }
> #endif

Python stores information in the thread-state structure which is 
specific to the thread - information such as the exception being 
handled, the current recursion depth, etc.

The assertion above is reporting the fact that 2 distinct ThreadState 
objects are trying to be used on a single thread.  The same thread for 
which you are trying to create a thread-state already has such a state.

Earlier versions of Python made no attempt to detect this, but as far as 
everyone can tell, was always an implicit assumption.

Unfortunately, much of this stuff has never been thought through 
correctly.  Trying to work with multiple InterpreterState objects is 
also very difficult, and in some cases simply does not work, as not all 
global variables are stored in an InterpreterState.  Theoretically 
though, this is probably what you want - if a different InterpreterState 
is current, I would expect a ThreadState specific to the 
InterpreterState could be used - but I doubt it works <wink>

The easy answer is to stop trying to create multiple interpreter states, 
then use the PyGILState calls to manage your thread-state.  These should 
be 2 lines per function (rather than the many that exist now).  If you 
need to build in both pre 2.3 and 2.3+:
#if (PY_VERSION_HEX >= 0x02030000)
#define USE_GILSTATE
#endif
/* Then */
#ifdef USE_GILSTATE
   PyGILState_State state = PyGILState_Ensure();
#else
   20 lines of existing code
#endif
// your body, and at the end
#ifdef USE_GILSTATE
   PyGILState_Release(state);
#else
   existing code
#endif

Mark.