[New-bugs-announce] [issue37186] Everyone uses GIL wrong! = DEADLOCK
Roffild
report at bugs.python.org
Thu Jun 6 20:03:04 EDT 2019
New submission from Roffild <roffild at gmail.com>:
Everyone uses GIL wrong! = DEADLOCK
I used sub-interpreters in embedded Python:
https://github.com/Roffild/RoffildLibrary/blob/35ef39fafc164d260396b39b28ff897d44cf0adb/Libraries/Roffild/PythonDLL/private.h#L44
https://github.com/Roffild/RoffildLibrary/blob/35ef39fafc164d260396b39b28ff897d44cf0adb/Libraries/Roffild/PythonDLL/mql_class.c#L142
PyEval_AcquireThread(__interp->interp);
...
PyGILState_Ensure() = DEADLOCK
...
PyEval_ReleaseThread(__interp->interp);
A deadlock happens in the line:
https://github.com/python/cpython/blob/7114c6504a60365b8b0cd718da0ec8a737599fb9/Python/pystate.c#L1313
Of course in the help there is the note:
Note that the PyGILState_() functions assume there is only one global interpreter (created automatically by Py_Initialize()). Python supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_() API is unsupported.
But functions PyGILState_() are used in third-party libraries. Most often, these functions are used without checking that GIL is already locked. Often, these functions are added to the code for reinsurance only and this can affect performance.
Numpy:
https://github.com/numpy/numpy/blob/2d4975e75c210202293b894bf98faf12f4697a31/numpy/core/include/numpy/ndarraytypes.h#L987
https://github.com/numpy/numpy/search?q=NPY_ALLOW_C_API&unscoped_q=NPY_ALLOW_C_API
Pytorch:
https://github.com/pytorch/pytorch/blob/0a3fb45d3d2cfacbd0469bbdba0e6cb1a2cd1bbe/torch/csrc/utils/auto_gil.h#L9
https://github.com/pytorch/pytorch/search?q=AutoGIL&unscoped_q=AutoGIL
Pybind11 developers have already fixed this problem:
https://github.com/pybind/pybind11/blob/97784dad3e518ccb415d5db57ff9b933495d9024/include/pybind11/pybind11.h#L1846
It is necessary to change the code of PyGILState_() functions to support sub-interpreters.
Or add to https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock warning:
Some Python libraries cannot be used in a sub-interpreter due to the likelihood of deadlock.
For me, this is a critical vulnerability!
There is another problem:
Calling PyEval_AcquireThread() again results in a deadlock. This can be controlled in your code, but not in a third-party library.
----------
components: Interpreter Core
messages: 344891
nosy: Roffild
priority: normal
severity: normal
status: open
title: Everyone uses GIL wrong! = DEADLOCK
type: crash
versions: Python 3.7
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37186>
_______________________________________
More information about the New-bugs-announce
mailing list