Parallel Python

[Duncan Booth]

Laszlo Nagy <gandalf at designaproduct.biz> wrote:

> parallelpython at gmail.com wrote:
>> Has anybody tried to run parallel python applications?
>> It appears that if your application is computation-bound using 'thread'
>> or 'threading' modules will not get you any speedup. That is because
>> python interpreter uses GIL(Global Interpreter Lock) for internal
>> bookkeeping. The later allows only one python byte-code instruction to
>> be executed at a time even if you have a multiprocessor computer.
>> To overcome this limitation, I've created ppsmp module:
>> http://www.parallelpython.com
>> It provides an easy way to run parallel python applications on smp
>> computers.
>> I would appreciate any comments/suggestions regarding it.
>>   
> I always thought that if you use multiple processes (e.g. os.fork) then 
> Python can take advantage of multiple processors. I think the GIL locks 
> one processor only. The problem is that one interpreted can be run on 
> one processor only. Am I not right? Is your ppm module runs the same 
> interpreter on multiple processors? That would be very interesting, and 
> something new.
> 
The GIL locks all processors, but just for one process. So, yes, if you 
spawn off multiple processes then Python will take advantage of this. For 
example we run Zope on a couple of dual processor dual core systems, so we 
use squid and pound to ensure that the requests are spread across 4 
instances of Zope on each machine. That way we do get a fairly even cpu 
usage.

For some applications it is much harder to split the tasks across separate 
processes rather than just separate threads, but there is a benefit once 
you've done it since you can then distribute the processing across cpus on 
separate machines.

The 'parallel python' site seems very sparse on the details of how it is 
implemented but it looks like all it is doing is spawning some subprocesses 
and using some simple ipc to pass details of the calls and results. I can't 
tell from reading it what it is supposed to add over any of the other 
systems which do the same.

Combined with the closed source 'no redistribution' license I can't really 
see anyone using it.