[Image-SIG] Multi-threading and PIL

[Fredrik Lundh]

Kevin Cazabon wrote:

> A couple years ago I did some work on enabling multi-threading in the 
> PIL core library, with the help of Fredrik and others.  We successfully 
> implemented it by releasing the GIL prior to starting many of the larger 
> routines/functions in the C library, and re-acquiring it before handing 
> back to the wrappers.  I can't remember exactly which functions we did 
> this to in total, but it definitely included the Image.resize code and 
> so forth.  On my dual-processor machine, I saw almost double performance 
> on multi-threaded resizing operations - proving it worked.  However, we 
> never touched the pixel access sections.
> 
> You could look at the resize code in the C module and use the same 
> method to try it with the functions you use most.  The trick is just 
> figuring out the right places to release and recapture the GIL.

The fact that the OP uses pixel access pretty much implies that most of 
the processing time in his program is spent on the Python level, which 
means that the GIL is the culprit here.  Releasing and reacquiring the 
lock would most likely take a lot more time than just fetching the pixel.

To fully get around the GIL for cases like this, you have to use 
processing or similar libraries.  For some cases, the simple 
thread/subprocess approach outlined here:

     http://effbot.org/zone/wide-finder.htm

can work (see the addenda section for examples using the processing and 
pprocess libraries).

Depending on the access/crunching ratio, psyco might also help:

     http://psyco.sourceforge.net/

</F>