Threading Question

[David Bolen]

David Lees <Davidl at niospammyraqia.com> writes:

> The code I am actually working on is for handeling sockets.  I am pretty
> careful and am passing things in a threadsafe manner between them.  Is
> there any way I can release the lock without writing C code?

If you mean within a single application, then not that I'm aware of.

As pointed out elsewhere in the thread, the lock is already being
released by any blocking operations, so you are getting overlapped
processing with respect to your socket I/O.  However, while that can
benefit overall efficiency of an application, since actual
interpretation of Python code (and thus any computation coded in
Python) is synchronized by the global interpreter lock, you can't get
CPU sharing benefit.

If you want to stick with pure Python, you might consider separating
your computation or CPU heavy tasks into independent modules that
communicate amongst themselves on the single system.  You could pass
computation data off to helper processes (instead of threads), which
then as independent Python processes could execute simultaneously.
There are plenty of communication mechanisms, but it might be as
simple as pickling the input and result data and shipping them over a
pipe between the processes.

Alternatively if you've got some clear CPU intensive stuff, it might
also be just as easy to make a small extension module (but yes, then
you'd be in C code) just with those functions that would release the
pinterpreter lock during their computation.

I suppose it might be interesting to try to create an extension module
whose purpose was to create new interpreters at the C level for
parallel execution.  You'd still have the issue of communicating a
request, but would stay in a single process and could remain as Python
code.

-- David