Parallel Python

[robert]

Duncan Booth wrote:
> Laszlo Nagy <gandalf at designaproduct.biz> wrote:
> 
> 
> The 'parallel python' site seems very sparse on the details of how it is 
> implemented but it looks like all it is doing is spawning some subprocesses 
> and using some simple ipc to pass details of the calls and results. I can't 
> tell from reading it what it is supposed to add over any of the other 
> systems which do the same.
> 
> Combined with the closed source 'no redistribution' license I can't really 
> see anyone using it.


Thats true. IPC through sockets or (somewhat faster) shared memory -  cPickle at least - is usually the maximum of such approaches.
See http://groups.google.de/group/comp.lang.python/browse_frm/thread/f822ec289f30b26a

For tasks really requiring threading one can consider IronPython.
Most advanced technique I've see for CPython ist posh : http://poshmodule.sourceforge.net/ 

I'd say Py3K should just do the locking job for dicts / collections, obmalloc and refcount (or drop the refcount mechanism) and do the other minor things in order to enable free threading. Or at least enable careful sharing of Py-Objects between multiple separated Interpreter instances of one process.
.NET and Java have shown that the speed costs for this technique are no so extreme. I guess less than 10%. 
And Python is a VHLL with less focus on speed anyway.
Also see discussions in http://groups.google.de/group/comp.lang.python/browse_frm/thread/f822ec289f30b26a .


Robert