Extension modules, Threading, and the GIL

[Donn Cave]

Quoth David Abrahams <dave at boost-consulting.com>:
| "Donn Cave" <donn at u.washington.edu> writes:
...
|> But while I'm not opposed to some brilliant fix to the thread
|> bootstrapping problems in Python, my simplistic answer would be that
|> external function calls should always release the interpreter lock,
|> whether they need to in anticipation of callbacks or not.  
|
| Interesting; that's what the author of B said as well.  However, most
| of my users have absolutely no need to release the GIL, and I'd hate
| to make them pay for it just because a few strange applications do.
|
|> The only exception would be the kind of thing that is really
|> transparently free of any kind of I/O, recursion 
|
| What does recursion have to do with anything?
|
|> or nontrivial computation.  Not only because of the present issue,
|> but to make the most of whatever concurrency you have in Python,
|> which is of course limited to these moments.  
|
| I understand the sentiment.  However, lots of Python applications
| aren't using threading at all, so it seems like a bit of a waste to
| manipulate the GIL at all in those cases.
|
| Hm. I think I take it back.  If the computation is really nontrivial,
| we can afford to waste a little GIL manipulation ;-)

If it helps to think about how often this already happens in the
interpreter, check out posixmodule.c, where calls to things like
uname() release the lock.  I don't know how uname() could take any
appreciable time to complete, but the philosophy seems to be that
it's not worth guessing about it, better to just drop the lock.

I should have said "callbacks" instead of "recursion".  I was thinking
that it might be an accepted usage, and that it might help matters to
think about it that way.  While we tend to use words like "asynchronous"
in connection with callback programming, of course it's normally just
a dispatch loop that has been implemented outside the application,
processing events in some order and calling back, all in the same
thread of control and all quite synchronous in that broader context
that includes the external dispatcher.  But I meant callbacks,
recursion only in the sense of calling back to the same application
component and not necessarily the exact same function.

|> Then the callback's problem is to find out which thread it belongs
|> to - at least, in my C++ wrappers I do undertake to always acquire
|> the same thread context for the same thread, and I think that's best
|> if you want exceptions and so forth to work correctly.  It's easy in
|> my application context, but I can see how it would be a problem in a
|> more generic situation, and a thread:threadstate table might be a
|> handy addition to the core.
|
| How do you approach the problem of figuring out which thread context
| to acquire?

Well, it's easy but probably irrelevant.  The system dispatching goes
through a C++ class with your usual virtual function callbacks, and in
any case where I expect to support the callbacks, I also create the object,
and in so doing its thread and thread state.  I use my own subclass with
the thread state included, so it's right there for the virtual function. 

Actually most of the dispatching objects don't have their own thread,
but they're attached to a dispatching thread object and know how to find
it -- specifically, for example, "view" graphics objects attached to a
"window" thread.  This is the BeOS native API.

	Donn Cave, donn at u.washington.edu