Extension modules, Threading, and the GIL

[Bengt Richter]

On Tue, 31 Dec 2002 21:28:19 +0100, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:

>Bengt Richter wrote:
>>>David is right that it does not matter for this issue how a thread is 
>>>created: Python can deal with threads created through C API just as fine 
>>>as with "Python threads"; Python threads are created through the very 
>>>same C API.
>> 
>> Well, I meant if C stuff didn't use Python's C API to create a thread and
>> all the good python stuff that the thread or threading modules will do for you.
>> I meant C calling something like win32 API
>
>And I meant the very same thing. There is absolutely nothing wrong in 
>creating a thread thread CreateThread, and then using a Python 
>interpreter in it.
I realize, but  I don't think we are talking about the same thing.
The key is how you go about "using a Python interpreter in it".

In order to become a thread with full Python baggage, IWT you have to do the
equivalent of what thread.start_new_thread does under the hood. Somewhere there's
a python thread state as passed in threadmodule.c and pystate.c etc.

	tstate = PyThreadState_New(boot->interp);
	PyEval_AcquireThread(tstate);

Just calling the win32 CreateThread is not going to get you a tstate, AFAICS.

So this gives us two kinds of thread to discuss, distinguished by their having
an associated tstate or not. For the sake of brevity, perhaps we can call one
a "Python thread" and the other an "OS thread" realizing that underneath the Python
thread there is an OS thread carrying the baggage (not going into implementations that
migh do Python threads with its own scheduler/dispatcher using only one OS thread).

I was thinking Dave's problem might have to do with an OS thread separately spawned
and not having acquired the proper full baggage to swap tstate with a python thread,
yet somehow using someone else's baggage to use the interpreter.

>
>
>> and doing something totally off to the side as far as Python was concerned.
>> IWT the result of the above would be different from a C API call to e.g.,
>> 
>>     thread.start_new_thread(function, args[, kwargs]) 
>
>It would not be different at all. start_new_thread uses CreateThread 
>under the hoods.
Yes, but the converse is not true. CreateThreads doesn't use start_new_thread
under the hood, so though the OS threading is fundamentally the same, one is
a Python thread with special python-realated state and the other is not, UIAM.
>
>> Well, what would happen if a win32 thread that had not been started with
>> thread.start_new_thread or C-API equivalent, nevertheless attempted to
>> acquire the lock? 
>
>It would work just fine. The thread would either acquire it immediately, 
>or block until the thread that currently holds it releases it.
PyEval_AcquireThread(tstate) requires a tstate to swap in when acquiring the lock,
AFAICS, and a raw OS thread would not have a tstate unless it built one. I was trying
to suggest that there might be a thread in Dave's universe that doesn't have a tstate
built, and therefore can't properly acquire the lock. That's why I was musing on the
possibility of an OS thread automatically having a tstate built for it if it didn't
have one.

Regards,
Bengt Richter