[New-bugs-announce] [issue29881] Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit

[STINNER Victor]

New submission from STINNER Victor:

When I read Serhiy Storshaka's idea in issue #29878, it recalled me an old idea of writing a generalization of the _Py_IDENTIFIER() API.

_Py_IDENTIFIER is an API to initialize a static string variable once. The _Py_Identifier structure has a "next" field to create a single-linked chained list. It allows to clear all variables at exit in _PyUnicode_ClearStaticStrings().

I propose a similar API but for any PyObject* object, to be able to clear all static variables at exit. It should help to release all memory in Py_Finalize() and have a safer Python finalization.

See attached pull request for the API itself.

"Static variables" in C are variables with a limited scope: a single C file or a single function.

It seems like the API can remove some lines of code. Example of patch:

@@ -1452,14 +1450,14 @@ compiler_mod(struct compiler *c, mod_ty mod)
 {
     PyCodeObject *co;
     int addNone = 1;
-    static PyObject *module;
-    if (!module) {
-        module = PyUnicode_InternFromString("<module>");
-        if (!module)
-            return NULL;
+    _Py_STATICVAR(module);
+
+    if (_PY_STATICVAR_INIT(&module, PyUnicode_InternFromString("<module>"))) {
+        return 0;
     }
+
     /* Use 0 for firstlineno initially, will fixup in assemble(). */
-    if (!compiler_enter_scope(c, module, COMPILER_SCOPE_MODULE, mod, 0))
+    if (!compiler_enter_scope(c, module.obj, COMPILER_SCOPE_MODULE, mod, 0))
         return NULL;
     switch (mod->kind) {
     case Module_kind:

--

Drawbacks of the API:

* It adds one pointer per static variables, so increase the memory footprint of 8 bytes per variable
* It requires to write "var.obj" instead of just "var" to access the Python object


The API doesn't try to remove duplicated objects. I consider that it's not an issue since functions like PyLong_FromLong(<small int>) and PyUnicode_InternFromString("c string") already do it for us. Some functions create mutable variables like PyImport_Import() which creates an empty list.

--

Note: Eric Snow proposed a solution "solving multi-core Python":

* https://mail.python.org/pipermail/python-ideas/2015-June/034177.html
* http://ericsnowcurrently.blogspot.fr/2016/09/solving-mutli-core-python.html

I'm not sure if this API would help or not to implement such idea, but Eric's project is experimental and wasn't taken in account when designing the API.

----------
components: Interpreter Core
messages: 289999
nosy: haypo
priority: normal
severity: normal
status: open
title: Add a new private API for "static C variables" (_PyStaticVar) to clear them at exit
type: resource usage
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue29881>
_______________________________________