Extension modules, Threading, and the GIL

[Donn Cave]

Quoth David Abrahams <dave at boost-consulting.com>:
...
| This is the continuation of a thread originated on python-dev; I was
| asked to re-raise it here on python-list until the issue is better
| understood.
|
| The original posting was here:
|
|   http://aspn.activestate.com/ASPN/Mail/Message/python-dev/1482879
|
| The he essence of the problem is that there's no way to write code
| that uses the Python 'C' API and which has no knowledge of whether it
| is running on Python's main thread when it is entered.
|
| The two respondents were left with some questions; you can read those,
| and my responses, in the thread at the bottom of the page referenced
| above.

I'll agree with Martin von Loewis that it doesn't sound like you really
should care precisely whether it's the ``main'' thread or not, and it
would be interesting to read more about the application context here
with respect to the callbacks and threads, because it isn't that clear
why this problem actually comes up.

But while I'm not opposed to some brilliant fix to the thread bootstrapping
problems in Python, my simplistic answer would be that external function
calls should always release the interpreter lock, whether they need to in
anticipation of callbacks or not.  The only exception would be the kind of
thing that is really transparently free of any kind of I/O, recursion or
nontrivial computation.  Not only because of the present issue, but to make
the most of whatever concurrency you have in Python, which is of course
limited to these moments.  If implementation A fails to do this and it
causes problems for B, then it's time to fix A.

Then the callback's problem is to find out which thread it belongs to -
at least, in my C++ wrappers I do undertake to always acquire the same
thread context for the same thread, and I think that's best if you want
exceptions and so forth to work correctly.  It's easy in my application
context, but I can see how it would be a problem in a more generic
situation, and a thread:threadstate table might be a handy addition
to the core.

	Donn Cave, donn at u.washington.edu