[Python-checkins] GH-104668: Don't call PyOS_* hooks in subinterpreters (GH-104674)

[brandtbucher]

https://github.com/python/cpython/commit/357bed0bcd3c5d7c4a8caad451754a9a172aca3e
commit: 357bed0bcd3c5d7c4a8caad451754a9a172aca3e
branch: main
author: Brandt Bucher <brandtbucher at microsoft.com>
committer: brandtbucher <brandtbucher at gmail.com>
date: 2023-05-22T19:34:34Z
summary:

GH-104668: Don't call PyOS_* hooks in subinterpreters (GH-104674)

files:
A Misc/NEWS.d/next/C API/2023-05-19-10-22-34.gh-issue-104668.MLX1g9.rst
M Doc/c-api/veryhigh.rst
M Doc/whatsnew/3.12.rst
M Parser/myreadline.c

diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index 513856d8a48d..000a2d3d8790 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -167,6 +167,10 @@ the same library that the Python runtime is using.
    event loops, as done in the :file:`Modules/_tkinter.c` in the
    Python source code.
 
+   .. versionchanged:: 3.12
+      This function is only called from the
+      :ref:`main interpreter <sub-interpreter-support>`.
+
 
 .. c:var:: char* (*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, const char *)
 
@@ -187,6 +191,10 @@ the same library that the Python runtime is using.
       :c:func:`PyMem_RawRealloc`, instead of being allocated by
       :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`.
 
+   .. versionchanged:: 3.12
+      This function is only called from the
+      :ref:`main interpreter <sub-interpreter-support>`.
+
 .. c:function:: PyObject* PyRun_String(const char *str, int start, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyRun_StringFlags` below, leaving
diff --git a/Doc/whatsnew/3.12.rst b/Doc/whatsnew/3.12.rst
index efbd2ca3de12..4ff90664bb79 100644
--- a/Doc/whatsnew/3.12.rst
+++ b/Doc/whatsnew/3.12.rst
@@ -1476,6 +1476,15 @@ Porting to Python 3.12
   Note that :c:func:`PyType_FromMetaclass` (added in Python 3.12)
   already disallows creating classes whose metaclass overrides ``tp_new``.
 
+* :c:var:`PyOS_InputHook` and :c:var:`PyOS_ReadlineFunctionPointer` are no
+  longer called in :ref:`subinterpreters <sub-interpreter-support>`. This is
+  because clients generally rely on process-wide global state (since these
+  callbacks have no way of recovering extension module state).
+
+  This also avoids situations where extensions may find themselves running in a
+  subinterpreter that they don't support (or haven't yet been loaded in). See
+  :gh:`104668` for more info.
+
 Deprecated
 ----------
 
diff --git a/Misc/NEWS.d/next/C API/2023-05-19-10-22-34.gh-issue-104668.MLX1g9.rst b/Misc/NEWS.d/next/C API/2023-05-19-10-22-34.gh-issue-104668.MLX1g9.rst
new file mode 100644
index 000000000000..7b882afd7f81
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2023-05-19-10-22-34.gh-issue-104668.MLX1g9.rst	
@@ -0,0 +1,5 @@
+Don't call :c:var:`PyOS_InputHook` or :c:var:`PyOS_ReadlineFunctionPointer`
+in subinterpreters, since it's generally difficult to avoid using global
+state in their registered callbacks. This also avoids situations where
+extensions may find themselves running in a subinterpreter they don't
+support (or haven't yet been loaded in).
diff --git a/Parser/myreadline.c b/Parser/myreadline.c
index 3f0e29f051a4..7074aba74b72 100644
--- a/Parser/myreadline.c
+++ b/Parser/myreadline.c
@@ -45,7 +45,10 @@ my_fgets(PyThreadState* tstate, char *buf, int len, FILE *fp)
 #endif
 
     while (1) {
-        if (PyOS_InputHook != NULL) {
+        if (PyOS_InputHook != NULL &&
+            // GH-104668: See PyOS_ReadlineFunctionPointer's comment below...
+            _Py_IsMainInterpreter(tstate->interp))
+        {
             (void)(PyOS_InputHook)();
         }
 
@@ -131,7 +134,10 @@ _PyOS_WindowsConsoleReadline(PyThreadState *tstate, HANDLE hStdIn)
     wbuf = wbuf_local;
     wbuflen = sizeof(wbuf_local) / sizeof(wbuf_local[0]) - 1;
     while (1) {
-        if (PyOS_InputHook != NULL) {
+        if (PyOS_InputHook != NULL &&
+            // GH-104668: See PyOS_ReadlineFunctionPointer's comment below...
+            _Py_IsMainInterpreter(tstate->interp))
+        {
             (void)(PyOS_InputHook)();
         }
         if (!ReadConsoleW(hStdIn, &wbuf[total_read], wbuflen - total_read, &n_read, NULL)) {
@@ -389,11 +395,23 @@ PyOS_Readline(FILE *sys_stdin, FILE *sys_stdout, const char *prompt)
      * a tty.  This can happen, for example if python is run like
      * this: python -i < test1.py
      */
-    if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout)))
-        rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt);
-    else
-        rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout,
-                                             prompt);
+    if (!isatty(fileno(sys_stdin)) || !isatty(fileno(sys_stdout)) ||
+        // GH-104668: Don't call global callbacks like PyOS_InputHook or
+        // PyOS_ReadlineFunctionPointer from subinterpreters, since it seems
+        // like there's no good way for users (like readline and tkinter) to
+        // avoid using global state to manage them. Plus, we generally don't
+        // want to cause trouble for libraries that don't know/care about
+        // subinterpreter support. If libraries really need better APIs that
+        // work per-interpreter and have ways to access module state, we can
+        // certainly add them later (but for now we'll cross our fingers and
+        // hope that nobody actually cares):
+        !_Py_IsMainInterpreter(tstate->interp))
+    {
+        rv = PyOS_StdioReadline(sys_stdin, sys_stdout, prompt);
+    }
+    else {
+        rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout, prompt);
+    }
     Py_END_ALLOW_THREADS
 
     PyThread_release_lock(_PyOS_ReadlineLock);