[New-bugs-announce] [issue42671] Make the Python finalization more deterministic

[STINNER Victor]

New submission from STINNER Victor <vstinner at python.org>:

At exit, Python calls Py_Finalize() which tries to clear every single Python objects. The order is which Python objects are cleared is not fully deterministic. Py_Finalize() uses an heuristic to attempt to clear modules of sys.modules in the "best" order.

The current code creates a weak reference to a module, set sys.modules[name] to None, and then clears the module attribute if and only if the module object was not destroyed (if the weak reference still points to the module).

The problem is that even if a module object is destroyed, the module dictionary can remain alive thanks for various kinds of strong references to it.

Worst case example:
---
class VerboseDel:
    def __del__(self):
        print("Goodbye Cruel World")
obj = VerboseDel()

def func():
    pass

import os
os.register_at_fork(after_in_child=func)
del os
del func

print("exit")
---

Output:
---
$ python3.9 script.py
exit
---

=> The VerboseDel object is never destroyed :-( BUG!


Explanation:

* os.register_at_fork(after_in_child=func) stores func in PyInterpreterState.after_forkers_child -> func() is kept alive until interpreter_clear() calls Py_CLEAR(interp->after_forkers_child);

* func() has reference to the module dictionary

I'm not sure why the VerboseDel object is not destroyed.


I propose to rewrite the finalize_modules() to clear modules in a more deterministic order:

* start by clearing __main__ module variables
* then iterate on reversed(sys.modules.values()) and clear the module variables
* Module attributes are cleared by _PyModule_ClearDict(): iterate on reversed(module.__dict__) and set dict values to None


Drawback: it is a backward incompatible change. Code which worked by luck previously no longer works. I'm talking about applications which rely on __del__() methods being calling in an exact order and expect Python being in a specific state.

Example:
---
class VerboseDel:
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print(self.name)

a = VerboseDel("a")
b = VerboseDel("b")
c = VerboseDel("c")
---

Output:
---
c
b
a
---

=> Module attributes are deleted in the reverse order of their definition: the most recent object is deleted first, the oldest is deleted last.


Example 2 with 3 modules (4 files):
---
$ cat a.py 
class VerboseDel:
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print(self.name)

a = VerboseDel("a")


$ cat b.py 
class VerboseDel:
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print(self.name)

b = VerboseDel("b")


$ cat c.py 
class VerboseDel:
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print(self.name)

c = VerboseDel("c")


$ cat z.py 
import a
import b
import c
---

Output:
---
$ ./python z.py 
c
b
a
---

=> Modules are deleted from the most recently imported (import c) to the least recently imported module (import a).

----------
components: Interpreter Core
messages: 383265
nosy: vstinner
priority: normal
severity: normal
status: open
title: Make the Python finalization more deterministic
versions: Python 3.10

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue42671>
_______________________________________