[Python-checkins] gh-98253: Break potential reference cycles in external code worsened by typing.py lru_cache (#98591)

[AlexWaygood]

https://github.com/python/cpython/commit/c314198fadd3dc79ee15f375a409163d8fb33586
commit: c314198fadd3dc79ee15f375a409163d8fb33586
branch: main
author: Wenzel Jakob <wenzel.jakob at epfl.ch>
committer: AlexWaygood <Alex.Waygood at Gmail.com>
date: 2022-11-30T08:33:32Z
summary:

gh-98253: Break potential reference cycles in external code worsened by typing.py lru_cache (#98591)

files:
A Misc/NEWS.d/next/Library/2022-10-24-11-01-05.gh-issue-98253.HVd5v4.rst
M Lib/typing.py

diff --git a/Lib/typing.py b/Lib/typing.py
index 127e3c942d04..38e227e3c55d 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -325,6 +325,7 @@ def _flatten_literal_params(parameters):
 
 
 _cleanups = []
+_caches = {}
 
 
 def _tp_cache(func=None, /, *, typed=False):
@@ -332,13 +333,20 @@ def _tp_cache(func=None, /, *, typed=False):
     original function for non-hashable arguments.
     """
     def decorator(func):
-        cached = functools.lru_cache(typed=typed)(func)
-        _cleanups.append(cached.cache_clear)
+        # The callback 'inner' references the newly created lru_cache
+        # indirectly by performing a lookup in the global '_caches' dictionary.
+        # This breaks a reference that can be problematic when combined with
+        # C API extensions that leak references to types. See GH-98253.
+
+        cache = functools.lru_cache(typed=typed)(func)
+        _caches[func] = cache
+        _cleanups.append(cache.cache_clear)
+        del cache
 
         @functools.wraps(func)
         def inner(*args, **kwds):
             try:
-                return cached(*args, **kwds)
+                return _caches[func](*args, **kwds)
             except TypeError:
                 pass  # All real errors (not unhashable args) are raised below.
             return func(*args, **kwds)
diff --git a/Misc/NEWS.d/next/Library/2022-10-24-11-01-05.gh-issue-98253.HVd5v4.rst b/Misc/NEWS.d/next/Library/2022-10-24-11-01-05.gh-issue-98253.HVd5v4.rst
new file mode 100644
index 000000000000..00df0070f3b9
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2022-10-24-11-01-05.gh-issue-98253.HVd5v4.rst
@@ -0,0 +1,10 @@
+The implementation of the typing module is now more resilient to reference
+leaks in binary extension modules.
+
+Previously, a reference leak in a typed C API-based extension module could leak
+internals of the typing module, which could in turn introduce leaks in
+essentially any other package with typed function signatures. Although the
+typing package is not the original source of the problem, such non-local
+dependences exacerbate debugging of large-scale projects, and the
+implementation was therefore changed to reduce harm by providing better
+isolation.