python distributed computing

[julien godin]

Hello the list,

I have a, let say 'special' , request.
I would like to make a very specific type of distributed computing.
Let me explain myself :

I have a HUUUUUGE(everything is relative) amount of data coming from UDP/514
( good old syslog), and one process alone could not handle them ( it's
full of regex, comparison and counting. all this at about 100K lines of data
per seconds and each line is about 1500o long. I tried to handle it on one
process, 100% core time in htop. )

So i said to myself : why not distribute the computing ?
I looked for a good library that could do the distribution/process
management job for me (http://wiki.python.org/moin/ParallelProcessing).

What i found was not really usable by me ( distributed processing with a lot
of processing but almost no process communication wich is the opposite of
the thing i would like to do: distribute the events in real time so it is
processed in real time.).

My question is : is their any library or API or anything that is able to do
that ?

Sincerly
Julien

PS: This is my first time posting on any mailing list, my explanation/syntax
could look a little bite, messy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110803/0d1c170c/attachment.html>