[Python-Dev] Sub-interpreters: importing numpy causes hang

[Stephan Reiter]

Reading through that post, I think I have everything covered but this here:
- The third and final scenario, and the one where the extended GIL
state functions for Ensure is still required, is where code doesn't
have the GIL as yet and wants to make a call into sub interpreter
rather than the main interpreter, where it already has a pointer to
the sub interpreter and nothing more. In this case the new
PyGILState_EnsureEx() function is used, with the sub interpreter being
passed as argument.

If I understand it correctly, it means the following in practice:
Whenever I or a third-party library start a new thread, we need to
query what interpreter we are running at the moment (in the thread
that is starting the new thread) and pass that information on to the
new thread so that it can initialize the GIL for itself.

Pseudo code ahead:
void do_in_thread(func_t *what) {
  PyThreadState* state = PyThreadState_Get(); /// or new
PyInterpreterState_Current();
  PyInterpreterState *interpreter = state->interp;
  std::thread t([what, interpreter] {
    auto s = PyGILState_EnsureEx(interpreter);
    what();
    PyGILState_Release(s); // could also release before what() because
TLS was updated and next PyGILState_Ensure() will work
  });
}

Did I get that right?

Stephan

Am Mo., 28. Jan. 2019 um 09:27 Uhr schrieb Nick Coghlan <ncoghlan at gmail.com>:
>
> On Mon, 28 Jan 2019 at 00:32, Stephan Reiter <stephan.reiter at gmail.com> wrote:
> >
> > Cool. Thanks, Nick!
> >
> > I did experiments based on this idea (https://github.com/stephanreiter/cpython/commit/3bca91c26ac81e517b4aa22302be1741b3315622) and haven't rejected it yet. :-)
>
> After talking to Graham about this, I unfortunately realised that the
> reason the callback approach is appearing to work for you is because
> your application is single-threaded, so you can readily map any
> invocation of the callback to the desired interpreter. Multi-threaded
> applications won't have that luxury - they need to be able to set the
> callback target on a per-thread basis.
>
> Graham actually described a plausible approach for doing that several
> years back: https://bugs.python.org/issue10915#msg126387
>
> We have much better subinterpreter testing support now, so if this is
> any area that you're interested in, one potential place to start would
> be to get Antoine's patch back to a point where it applies and
> compiles again.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia