Extension modules, Threading, and the GIL

[Bengt Richter]

On Tue, 31 Dec 2002 18:24:33 +0100, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= <martin at v.loewis.de> wrote:

>Bengt Richter wrote:
>> Perhaps this discussion needs to distinguish clearly between OS
>> threads and Python threads? 
>
>David is right that it does not matter for this issue how a thread is 
>created: Python can deal with threads created through C API just as fine 
>as with "Python threads"; Python threads are created through the very 
>same C API.
Well, I meant if C stuff didn't use Python's C API to create a thread and
all the good python stuff that the thread or threading modules will do for you.
I meant C calling something like win32 API, e.g.,

HANDLE CreateThread(
    LPSECURITY_ATTRIBUTES  lpThreadAttributes,	// address of thread security attributes  
    DWORD  dwStackSize,	// initial thread stack size, in bytes 
    LPTHREAD_START_ROUTINE  lpStartAddress,	// address of thread function 
    LPVOID  lpParameter,	// argument for new thread 
    DWORD  dwCreationFlags,	// creation flags 
    LPDWORD  lpThreadId 	// address of returned thread identifier 
   );	

and doing something totally off to the side as far as Python was concerned.
IWT the result of the above would be different from a C API call to e.g.,

    thread.start_new_thread(function, args[, kwargs]) 

(whether via run simple string or whatever).
>> I read the stuff at the URL above (not saying I totally digested it)
>> and got the impression that Dave is saying a "callback" may be called
>> **without** the running and calling thread having acquired the GIL.
>
>I'm getting the feeling that the interpreter *lock* is irrelevant for 
>the problem at hand. The observed behaviour was not a deadlock, but that 
>Python suddenly had a NULL thread state. This is not supposed to happen, 
>and it is unclear to me why it happened.
>
Well, what would happen if a win32 thread that had not been started with
thread.start_new_thread or C-API equivalent, nevertheless attempted to
acquire the lock? In C IWT you might be able to bypass stuff you shouldn't,
and if the lock was aquired, switching the main thread out, what would get
switched in, since there would be no other thread.start_new_thread-created
python thread state to go to -- since by definition we are abusing things
with a raw win32 thread? Anyway, that's the kind of thread distinction I
was trying to talk about.

OTOH, could there be a race condition that gets to a thread-data reference
before it has been switched over from the previous owner of the lock? You'd
think that would show up more regularly, but sometimes weird convoying of
events happen to make things work a long time, until a random interrupt
hits in a couple-of-microseconds window.

Regards,
Bengt Richter