Bus error in PyGILState_Release (callbacks from other threads)

[geoffschmidt at gmail.com]

[Note: I don't check the mailbox in the header. Please send any
correspondence to the address listed below.]

I'm trying to write an extension in C that delivers callbacks to
Python. The C code starts several threads, and I'd like one of the new
threads that is started to be able to deliver callbacks to Python. I
thought I could do this by wrapping the callback function in
PyGILState_Ensure / PyGILState_Release. When I do this, the Python code
in the callback in between those two calls certainly works, but when
PyGILState_Release is called, the process dies with a bus error!

Is this supposed to work, or am I doing something terribly wrong? (Code
attached below.) This is the official Python 2.4.3 build for OS X; I'm
running OS X 10.4.6.

My examination of the source suggests that head_mutex in pystate.c is
becoming 0x20, which is not so good, since it is supposed to be a
pointer. (in pystate.c: PyGILState_Release calls
PyThreadState_DeleteCurrent, which calls tstate_delete_common, which
calls HEAD_UNLOCK, which calls PyThread_release_lock in
thread_pthread.h; this function's first action is to call
pthread_mutex_lock, which dies on a bus error because the value of
head_mutex that HEAD_UNLOCK got and passed to PyThread_release_lock
pointed off into outer space. This is educated guesswork -- gdb's not
reporting a whole lot in the call stack.) Interestingly, the prior call
to HEAD_LOCK in tstate_delete_common succeeds, so it sort of looks like
memory is being corrupted in between the HEAD_LOCK at around
pystate.c:245 and the HEAD_UNLOCK around pystate.c:254.

	HEAD_LOCK(); // <-- guess: this is succeeding
	for (p = &interp->tstate_head; ; p = &(*p)->next) {
		if (*p == NULL)
			Py_FatalError(
				"PyThreadState_Delete: invalid tstate");
		if (*p == tstate)
			break;
	}
	*p = tstate->next;
	HEAD_UNLOCK(); // <-- guess: this is resulting in a bus error

Here's some Pyrex code that crashes 100% of the time for me:

--- snip (foo.pyx) ---
cdef extern from "stdio.h":
    int printf(char *str, ...)

cdef extern from "Python.h":
    ctypedef int PyGILState_STATE
    PyGILState_STATE PyGILState_Ensure()
    void PyGILState_Release(PyGILState_STATE gstate)

cdef extern from "pthread.h":
    ctypedef void *pthread_t # it'll do
    int pthread_create(pthread_t *thread, void *attr,
                       void *(*start_routine)(void *), void *arg)

cdef extern void *func(void *x):
    printf("Entering func(%p)\n", x)

    cdef PyGILState_STATE st
    printf("PyGILState_Ensure\n")
    st = PyGILState_Ensure()
    printf("PyGILState_Release\n")
    PyGILState_Release(st)
    printf("Leaving func\n")

def callFuncDirectly():
    func(NULL)

def callFuncInThread():
    cdef pthread_t thr
    pthread_create(&thr, NULL, func, NULL);

--- snip (setup.py) ---

from distutils.core import setup
from distutils.extension import Extension
from Pyrex.Distutils import build_ext

setup(
    name = 'foo',
    ext_modules = [Extension("foo", ["foo.pyx"])],
    cmdclass = {'build_ext': build_ext},
    )

--- end ---

I ran 'python setup.py build_ext --inplace', then started python in
that directory, did 'import foo', and then 'foo.callFuncDirectly()'. No
crash. Then I called 'foo.callFuncInThread()'. Prints
'PyGILState_Release' and then falls over.

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000020
[Switching to process 25649 thread 0x313]
0x900017dc in pthread_mutex_lock ()
(gdb) bt
#0  0x900017dc in pthread_mutex_lock ()
#1  0x002be654 in PyThread_release_lock (lock=0x20) at
/Volumes/Data/Users/ronald/Universal/python24-fat/Python/thread_pthread.h:439
#2  0x00045600 in func (__pyx_v_x=0x0) at foo.c:72
#3  0x9002ba68 in _pthread_body ()

Any help (or even just confirmation along the lines of "this is
supposed to work, file a bug") would be greatly appreciated.

thanks,
Geoff Schmidt
gschmidt [[a t]] gschmidt [[d o t]] org (send correspondence here, not
the address in the header)