extension module, thread safety?

[David Bolen]

Torsten Mohr <tmohr at s.netic.de> writes:

> The question came up if this is by itself thread safe,
> if some two or more threads try to change these data types,
> are the C functions by themselves are "atomic" or can they
> be interrupted be the perl interpreter and then (data types
> are in some inconsistent half-changed state) another function
> that works on these data is called?

I presume you mean "Python" and not "perl"...

If the threads under discussion are all Python threads, then by
default yes, the extension module C functions will appear to be atomic
from the perspective of the Python code.  When the Python code calls
into the extension module, the GIL (global interpreter lock) is still
being held.  Unless the extension module code explicitly releases the
GIL, no other Python threads can execute (even though those threads
are in fact implemented as native platform threads).

So in general, if you write an extension module where none of its
functions ever release the GIL, there's no way for two of its
functions to be run from different Python threads simultaneously.

Note that this restriction won't necessarily hold if there are other
ways (at the C level, or from other extension modules) to trigger code
in the extension module, since that's outside of the control of the
Python GIL.  Nor will it necessarily hold true if your extension
module calls back out into Python (as a callback, or whatever) since
once the interpreter is back in Python code the interpreter itself
will periodically release the GIL, or some other extension code that
the callback code runs may release it.

To the extent possible, it's considered good practice to release the
GIL in an extension module whenever you are doing lengthy processing
so as to permit other Python threads (that may have nothing to do with
using your extension module) to execute.  For short routines this
really isn't an issue, but if your extension module will be spending
some time managing its data, you may wish to add some internal thread
protection around that data, so that you can use your own locks rather
than depending on the GIL.

-- David