[issue33608] [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
STINNER Victor
report at bugs.python.org
Tue Mar 5 06:23:11 EST 2019
STINNER Victor <vstinner at redhat.com> added the comment:
> That's okay, Victor. Thanks for jumping on this. I'll take a look when I get a chance.
>From what I saw, your first commit was enough to reproduce the crash.
If I recall correctly, Antoine Pitrou modified the GIL so threads exit immediately when Py_Finalize() is called. I'm thinking at:
void
PyEval_RestoreThread(PyThreadState *tstate)
{
...
take_gil(tstate);
/* _Py_Finalizing is protected by the GIL */
if (_Py_IsFinalizing() && !_Py_CURRENTLY_FINALIZING(tstate)) {
drop_gil(tstate);
PyThread_exit_thread();
Py_UNREACHABLE();
}
...
PyThreadState_Swap(tstate);
}
Problem: this code uses tstate, whereas the crash occurred because tstate pointed to freed memory:
"""
Thread 1 got the crash
(gdb) p *tstate
$3 = {
prev = 0xdbdbdbdbdbdbdbdb,
next = 0xdbdbdbdbdbdbdbdb,
interp = 0xdbdbdbdbdbdbdbdb,
...
}
...
Thread 1 (LWP 100696):
#0 0x0000000000368210 in take_gil (tstate=0x8027e2050) at Python/ceval_gil.h:216
#1 0x0000000000368a94 in PyEval_RestoreThread (tstate=0x8027e2050) at Python/ceval.c:281
...
"""
https://bugs.python.org/issue36114#msg337090
When this crash occurred, Py_Finalize() already completed in the main thread!
"""
void _Py_NO_RETURN
Py_Exit(int sts)
{
if (Py_FinalizeEx() < 0) { /* <==== DONE! */
sts = 120;
}
exit(sts); /* <=============== Crash occurred here! */
}
"""
Py_Finalize() is supposed to wait for threads before deleting Python thread states:
"""
int
Py_FinalizeEx(void)
{
...
/* The interpreter is still entirely intact at this point, and the
* exit funcs may be relying on that. In particular, if some thread
* or exit func is still waiting to do an import, the import machinery
* expects Py_IsInitialized() to return true. So don't say the
* interpreter is uninitialized until after the exit funcs have run.
* Note that Threading.py uses an exit func to do a join on all the
* threads created thru it, so this also protects pending imports in
* the threads created via Threading.
*/
call_py_exitfuncs(interp);
...
/* Remaining threads (e.g. daemon threads) will automatically exit
after taking the GIL (in PyEval_RestoreThread()). */
_PyRuntime.finalizing = tstate;
_PyRuntime.initialized = 0;
_PyRuntime.core_initialized = 0;
...
/* Delete current thread. After this, many C API calls become crashy. */
PyThreadState_Swap(NULL);
PyInterpreterState_Delete(interp);
...
}
"""
The real problem for years are *deamon threads* which... BY DESIGN... remain alive after Py_Finalize() exit! But as I explained, they must exit as soon at they attempt to get GIL.
----------
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue33608>
_______________________________________
More information about the Python-bugs-list
mailing list