[issue15751] Support subinterpreters in the GIL state API

[Nick Coghlan]

Nick Coghlan added the comment:

The reason I'm proposing going back to the original SwitchInterpreter idea is because the EnsureEx idea doesn't nest cleanly - if your thread is already associated with "interpreter A", you can't readily call into "interpeter B", because the API provides no way to correctly restore the associated interpreter back to interpreter A when you're done.

EnsureEx works fine if extension modules are not aware of multiple interpreters:

   /* Embedding application (GIL always unlocked) */
   gilstate = PyGILState_EnsureEx(interp_A);
     /* Python code and extension code happens */
       /* Callback needs to reach back into Python */
       cb_gilstate = PyGILState_Ensure();
         /* Callback runs */
       PyGILState_Release(cb_gilstate);
     /* Python code and extension code unwinds */
   PyGILState_Release(gilstate);

However, it *doesn't* work (at least, not easily) if the extension itself wants to call back into an interpreter other than the one already associated with the current thread:

   /* Embedding application (GIL always unlocked) */
   gilstate = PyGILState_EnsureEx(interp_A);
     /* Python code and extension code happens */
       /* Callback needs to reach back into a specific interpreter */
       cb_gilstate = PyGILState_EnsureEx(interp_B);
         /* Callback runs */
       PyGILState_Release(cb_gilstate);
     /* Python code and extension code unwinds */
   PyGILState_Release(gilstate);

Does that second call to EnsureEx fail? If it succeeds, how does the client know which interpreter to use for the PyGILState_Release call? It could be made to work if PyGILState_STATE was changed from an enum to a struct that included in interpreter state pointer, or if EnsureEx returned a different type and was paired up with a new ReleaseEx pointer.

However, that's starting to get very messy compared to a separate SwitchInterpreter call:

   /* Embedding application (GIL always unlocked) */
   old_interp = PyGILState_SwitchInterpreter(interp_A);
   /* "autoTLSkey" now refers to a thread state for interpreter A */
   gilstate = PyGILState_Ensure();
     /* Python code and extension code happens */
       /* Callback needs to reach back into Python */
       pre_cb_interp = PyGILState_SwitchInterpreter(interp_B);
       /* "autoTLSkey" now refers to a thread state for interpreter B */
       cb_gilstate = PyGILState_Ensure();
         /* Callback runs */
       PyGILState_Release(cb_gilstate);
       PyGILState_SwitchInterpreter(pre_cb_interp);
       /* "autoTLSkey" again refers to a thread state for interpreter A */
     /* Python code and extension code unwinds */
   PyGILState_Release(gilstate);
   PyGILState_SwitchInterpreter(old_interp);

And yes, I'm pondering ways that this could be used to implement Rust-style "channels" for communication between interpreters without needing to copy data, by using this API to create proxy interfaces for accessing an object owned by another subinterpreter.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue15751>
_______________________________________