C-extension not threadable?

[Dave Cole]

>>>>> "Martin" == Martin v Loewis <martin at v.loewis.de> writes:

Martin> stojek at part-gmbh.de (Marcus Stojek) writes:
>> The C-extension does what it is supposed to do, the callback
>> function works, but the C-function does not start as a parallel
>> thread but is freezing my UI. Anything basic I miss here ?

Martin> Yes, the Global Interpreter Lock (GIL). When C code runs that
Martin> was invoked from Python code, it holds the GIL, and only one
Martin> such C code can run at any time. So you need to release the
Martin> lock, using Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.

If the API which is being wrapped by the extension module utilises
callbacks, and you want to make those callbacks available to Python
code, and you want to do this in a threading environment where the GIL
is released when calling the API, then the above will not work.

I tried this for my Sybase module and received a segfault for my
troubles.

In cases like this you have to do something a teeny bit more
complicated.  In my case I was lucky in that the callback functions
from the Sybase library pass a pointer to a connection structure.  In
the callback I then use the pointer to locate my Python object which
wraps the Sybase connection structure where I store all sorts of
Python related info including:

1) A PyThread_type_lock lock which ensures that all thread
   access to a Sybase API's on a connection is serialised.

2) A PyThreadState* which stores the thread state of the thread
   currently operating on the connection.

Then each call to a connection related Sybase API (ct_cancel() for
example) where I want to release the GIL looks like this:

    SY_CONN_BEGIN_THREADS(self);
    status = ct_cancel(self->conn, NULL, type);
    SY_CONN_END_THREADS(self);

which expands (more or less) to this:

    PyThread_acquire_lock(self->lock, WAIT_LOCK);
    self->thread_state = PyEval_SaveThread();

    status = ct_cancel(self->conn, NULL, type);

    PyEval_RestoreThread(self->thread_state);
    self->thread_state = NULL;
    PyThread_release_lock(self->lock);

So when a get a callback on a connection all I have to do locate the
associated Python connection object (conn) and do this:

    PyEval_RestoreThread(conn->thread_state);
    conn->thread_state = NULL;
    :
    result = PyEval_CallObject(servermsg_cb, args);
    :
    conn->thread_state = PyEval_SaveThread();

When the Sybase API invokes the callback internal to my extension
module I reacquire the GIL with the correct thread state and then call
up to the Python callback function registered by the module user.
Once the Python callback function returns I release the GIL again
because it will be reacquired when the Sybase API returns (in this
case ct_cancel()).

This seems to be working fine so far (fingers crossed).

- Dave

-- 
http://www.object-craft.com.au