Threading problem

[Gustavo Cordova]

> 
> The Python global interpreter lock requires that even with native
> threading, only one thread is actually interpreting Python code at any
> moment in time.  The net result of this is that while threading is
> excellent for overlapping I/O, it won't efficiently distribute CPU
> amongst processors.  That's because if you are CPU bound, then you are
> stuck in Python code execution, and only one processor can be doing
> that at any given moment in time.
> 

This is evil :-(

>
> This can be improved somewhat if you use extensions that release the
> GIL, but in most cases that's done during I/O and not processing time,
> particularly if the latter will require access to Python structures.
> But in theory a purely CPU bound extension could release the GIL while
> performing computations that it knew didn't require any interface to
> Python.
> 

I suspected this.

So, a solution would be to make an extension that...

1. Copies all pertinent data from python-space into it's own memory.

2. Releases the GIL.

3. Does all the work that needs to be done only on it's private memory.

4. Recapture the GIL.

5. Copy back results into python-space.

Something like that?

-gustavo