c bindings with non-python thread callback while python exits

[Barry Scott]

> On 26 Jan 2021, at 14:35, Paul Grinberg <gri6507 at gmail.com> wrote:
> 
> I have a C++ library (luckily with source code, so I know what's going on) to which I wrote c bindings. The library internally starts an event pthread which generates callbacks back into python. The c binding for processing these callbacks look like this (simplified for convenience):
> 
> PyGILState_STATE gstate;
> gstate = PyGILState_Ensure();
> PyObject_CallObject(cb, arglist);
> PyGILState_Release(gstate);
> 
> This works perfectly while the python program using this c binding is running. However, when the program terminates normally and there happens to be one last callback underway, the c bindings hang. When viewed with GDB, I can see that the callback thread callstack looks like this:
> 
> futex_abstimed_wait_cancelable()
> _pthread_cond_wait_common()
> __pthread_cond_timedwait()
> PyCOND_TIMEDWAIT()
> take_gil()
> PyEval_RestoreThread()
> PyGILState_Ensure()
> c_bindings_callback()
> cpp_library_callback()
> 
> and the main thread callstack, which is where the python interpreter was running, looks like this
> 
> __pthread_clockjoin_ex ()
> std::thread::join()
> cpp_library_static_singleton_class_destructor ()
> __run_exit_handlers ()
> __GI_exit ()
> __libc_start_main ()
> _start ()
> 
> in other words, the program hangs because the main thread is waiting for the event pthread to join, but that thread is stuck in a callback waiting for the GIL.
> 
> What is the right way to prevent this problem from happening?

As you say you have dead locked.

My guess is that you have to shutdown your extension before you quit python.

You might find pycxx interesting if you are wrapping C++ code for use with python.

http://cxx.sourceforge.net/

Barry



> Thank you in advance,
> Paul.
> 
> P.S. I am running on Linux: ubuntu 18.04 with python 3.6.9, also reproduced with python 3.7.5, as well as ubuntu 20.04 with python 3.8.5
> -- 
> https://mail.python.org/mailman/listinfo/python-list
>