[New-bugs-announce] [issue46417] [subinterpreters] Clear static types in Py_Finalize()

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

Converting static types to heap types is a work-in-progress:

* bpo-40077: "Convert static types to heap types: use PyType_FromSpec()"
* At December 29, 2020, 43% (89/206) of types are declared as heap types on a total of 206 types. For comparison, in Python 3.8, only 9% (15/172) of types were declared as heap types: 74 types have been converted in the meanwhile.
* https://vstinner.github.io/isolate-subinterpreters.html

Static types are still causing issues with Py_Initialize() / Py_Finalize() is called multiple times in the same process, which is a supported use case for embedded Python. See bpo-45691 "Partial moving of core objects to interpreter state is incorrect at best, unsafe at worse" for a recent example with sys.float_info where sys.float_info.n_unnamed_fields holds a reference to a Python object.

In june 2020, I wrote GH-20763 to clear static types at exit. But I abandoned my attempt because of bugs. Copy of my https://bugs.python.org/issue1635741#msg371119 message:

"""
I wrote PR 20763 to "finalize" static types in Py_Finalize(). It mostly works, but "./Programs/_testembed test_forced_io_encoding" crash. This program calls Py_Initialize() and Py_Finalize() multiple times in a loop.

It doesn't look to be safe to clear static types. Many functions rely on the fact that static types are "always there" and are never finalized. Also, only a few static types are cleared by my PR: many static types are left unchanged. For example, static types of the _io module.

It seems like a safer approach is to continue the work on bpo-40077: "Convert static types to PyType_FromSpec()".
"""

I propose a "best effort" approach: only clear a static type if it is no longer used. For example, check its reference count and its __subclasses__() method.

----------
components: Subinterpreters
messages: 410808
nosy: vstinner
priority: normal
severity: normal
status: open
title: [subinterpreters] Clear static types in Py_Finalize()
versions: Python 3.11

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue46417>
_______________________________________