[issue15751] [subinterpreters] Make the PyGILState API compatible with subinterpreters

[Sebastian Berg]

Sebastian Berg <sebastian at sipsolutions.net> added the comment:

In NumPy ufuncs and datatype casting (iteration) we have the following setup:

   user-code (releasing GIL) -> NumPy API -> 3rd-party C-function 

(in the ufunc code, numpy is the one releasing the GIL, although others, such as numba probably hook in and replace that.)
Now, I want the 3rd-party C-function to be able to report errors in a reasonable way. Which, in my opinion means it must be able to grab the GIL, set an error (potentially release the GIL) and return an error result.
In theory, setting the error could be deferred to some later "cleanup" when the GIL is held, but that just does not seem practical to me.

One thing which may make this not as problematic is that the all of this can be expected to run within a Python created thread, so I somewhat think the initial proposal here would effectively fix the current NumPy usage (I am not sure).


The reason I ask/post this is, that this is all part of public-API which requires a complete re-implementation very soon. While extensions to that new API may be possible, that is always a bit painful, so it would be good to know how that API should be designed right now.

At this point, it seems that I should design the 3rd-party C-function API so that it is passed a `void *reserved = NULL` (always NULL) to be able to pass a `PyThreadState *` or `PyInterpreterState *` at some point safely. (Or a `PyThreadState **`/ `PyInterpreterState **`?)

In the majority of cases, I could already pass this right now, but I have currently no clear idea of what is the best practice. I also need to take such an argument (requiring new API) in at least one place. If we know how this will pan out, adding it sooner rather than later would be good, since it will make future transition/adoption faster or at least easier.

As far as I understand, right now PyGILState_Ensure()` not working is probably the single most blocking issue for correct subinterpreter support in NumPy, since this is baked into public API it may take years for true support in the above way, but we may be able to go a large part of the way now. My problem is that I can only do that if I am clear on what is needed.

----------
nosy: +seberg

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue15751>
_______________________________________