[Python-Dev] Cloning threading.py using proccesses

[Josiah Carlson]

"Richard Oudkerk" <r.m.oudkerk at googlemail.com> wrote:
> I am not sure how sensible the idea is, but I have had a first stab at
> writing a module processing.py which is a near clone of threading.py
> but uses processes and sockets for communication.  (It is one way of
> avoiding the GIL.)

On non-windows platforms, you should check on unix domain sockets, I've
found they can run a couple times faster than standard sockets on the
local machine.  And if you are using fork or a variant of subprocess to
start processes on linux or Windows, you should consider using pipes,
they can be competitive with sockets (though using a bunch on Windows
can be a pain).

> I have tested it on unix and windows and it seem to work pretty well.
> (Getting round the lack of os.fork on windows is a bit awkward.)
> There is also another module dummy_processing.py which has the same
> api but is just a wrapper round threading.py.
> 
> Queues, Locks, RLocks, Conditions, Semaphores and some other shared
> objects are implemented.
> 
> People are welcome to try out the tests in test_processing.py
> contained in the zipfile.  More information is included in the README
> file.
> 
> As a quick example, the code

[snip]

Looks interesting.  Maybe it would become clearer with docs (I hope
you've written some).  Right now there is a difference, and it is
basically that there are tokens and proxies, which could confuse some
users.

Presumably with this library you have created, you have also written a
fast object encoder/decoder (like marshal or pickle).  If it isn't any
faster than cPickle or marshal, then users may bypass the module and opt
for fork/etc. + XML-RPC; which works pretty well and gets them
multi-machine calling, milti-language interoperability, and some other
goodies, though it is a bit slow in terms of communication.


 - Josiah