Extension modules, Threading, and the GIL

[Greg Chapman]

On 01 Jan 2003 07:42:51 +0100, martin at v.loewis.de (Martin v. Löwis) wrote:

>bokr at oz.net (Bengt Richter) writes:
>
>> PyEval_AcquireThread(tstate) requires a tstate to swap in when
>> acquiring the lock, AFAICS, and a raw OS thread would not have a
>> tstate unless it built one. I was trying to suggest that there might
>> be a thread in Dave's universe that doesn't have a tstate built, and
>> therefore can't properly acquire the lock. That's why I was musing
>> on the possibility of an OS thread automatically having a tstate
>> built for it if it didn't have one.
>
>This is unlikely. tstate isn't a global variable, so if you invoke
>AcquireThread, you have to create a thread state yourself first. This
>is independent of whether the thread was created with
>start_new_thread: When you are in a position to invoke AcquireThread,
>you *never* have a thread state you could use, you always have to
>create one (unless you can come up with some clever machinery, such as
>storing the thread state in TLS). People using AcquireThread usually
>know this.
>

It seems to me that threadstate could be made global, or at least thread local,
within Python, thus freeing client code from ever having to explicitly create a
threadstate.  For this to work, Python would have to have the equivalent of
thread local storage on all supported platforms.  Looking over the thread-sig
archives, Greg Stein suggested that TLS could be emulated on platforms which
don't offer it natively using a Python dict and a lock (at any rate, it should
be possible with some sort of synchronized data structure).  With only one
threadstate per thread, a thread could easily determine whether it has the GIL
(the threadstate could have some sort of active flag which gets set when it
obtains the GIL); this might solve David Abraham's problem (not sure).  (It
could also allow a thread to call AcquireThread multiple times without deadlock;
since there would be only one threadstate per thread, that state could preserve
a lockcount to handle recursive calls.)

Thinking further about this, for this to work cleanly I think Python would have
to allow only one interpreter per process.  I never use multiple interpreters,
so I'm not quite sure what they're used for, but I wonder if the need for them
could be eliminated by providing a new built-in type (sort of like RExec without
the security overhead) which would initialize itself by doing the stuff that
Py_NewInterpreter does to get a new copy of the global data space and which
would provide methods for executing code in that copy of the data space.


---
Greg Chapman