[New-bugs-announce] [issue26003] Issues with PyEval_InitThreads and PyGILState_Ensure

[tzickel]

New submission from tzickel:

A few issues regarding threads:
A. (Python 2 & 3) The documentation (https://docs.python.org/3/c-api/init.html) about initializing the GIL/Threading system does not specify that calling PyEval_InitThreads actually binds the calling thread as the main_thread in the ceval.c, meaning that the thread will be in charge till the process goes down for handling Py_AddPendingCall calls, and if it ends/dies, they won't be handled anymore.

This ceval.c's main_thread is different BTW from the one in signalmodule.c which is bound to the thread that called Py_InitializeEx.

Maybe there is sense for both main_thread to be the same one and initialized in the same time ? (even without a GIL existing)

B. (Python 3) Besides the bad documentation regarding this, python 3.4 added issue #19576 which actually hides the call for PyEval_InitThreads inside PyGILState_Ensure. Without careful care and knowledge by the programmer, this might cause for a short lived thread created in C to actually bind the ceval.c's main_thread and when the thread dies main_thread will never be changed again.

The reason this is important is beforehand, the programmer needed to think about PyEval_InitThreads now it's hidden and not even mentioned in the documentation.

C. (Python 2 & 3) In PyEval_InitThreads documentation it's written "It is not safe to call this function when it is unknown which thread (if any) currently has the global interpreter lock." Thus it should be mentioned that PyGILState_Ensure is now calling it in the documentation ?

Also I believe the reason this quote exists is because a potential race condition between thread A which might be running code in PyEval_EvalFrameEx (before PyEval_InitThreads is called, and thus is not GIL aware), and thread B which calls PyEval_InitThreads then calls PyGILState_Ensure, then running Python code, while thread A is still running python code as well. I think it should be explained more clearly in the documentation the implications (race condition).

I think there might be a way to make an PyEval_InitThreads variant which can overcome this race condition. Basically it involves using Py_AddPendingCall to a C function which calls PyEval_InitThreads, and notifies the calling command/thread when it's done. This way we can be sure that the GIL is taken by one thread, and all the others are blocked. (maybe a signal should be sent as well, in case the main_thread is blocked on an I/O operation).

D. (Python 2) If the main_thread finishes it's job, while other python threads are still alive, signal handling isn't processed anymore (Example will be added as a file).

----------
components: Interpreter Core
files: signalexample.py
messages: 257425
nosy: tzickel
priority: normal
severity: normal
status: open
title: Issues with PyEval_InitThreads and PyGILState_Ensure
versions: Python 2.7, Python 3.6
Added file: http://bugs.python.org/file41485/signalexample.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue26003>
_______________________________________