[IPython-dev] ZMQ Parallel IPython Performance preview

[MinRK]

I have my first performance numbers for throughput with the new parallel
code riding on ZeroMQ, and results are fairly promising.  Roundtrip time for
~512 tiny tasks submitted as fast as they can is ~100x faster than with
Twisted.

As a throughput test, I submitted a flood of many very small tasks that
should take ~no time:
new-style:
def wait(t=0):
    import time
    time.sleep(t)
submit:
client.apply(wait, args=(t,))

Twisted:
task = StringTask("import time; time.sleep(%f)"%t)
submit:
client.run(task)

Flooding the queue with these tasks with t=0, and then waiting for the
results, I tracked two times:
Sent: the time from the first submit until the last submit returns
Roundtrip: the time from the first submit to getting the last result

Plotting these times vs number of messages, we see some decent numbers:
* The pure ZMQ scheduler is fastest, 10-100 times faster than Twisted
roundtrip
* The Python scheduler is ~3x slower roundtrip than pure ZMQ, but no penalty
to the submission rate
* Twisted performance falls off very quickly as the number of tasks grows
* ZMQ performance is quite flat

Legend:
zmq: the pure ZMQ Device is used for routing tasks
lru/weighted: the simplest/most complicated routing schemes respectively in
the Python ZMQ Scheduler (which supports dependencies)
twisted: the old IPython.kernel

[image: roundtrip.png]
[image: sent.png]
Test system:
Core-i7 930, 4x2 cores (ht), 4-engine cluster all over tcp/loopback, Ubuntu
10.04, Python 2.6.5

-MinRK
http://github.com/minrk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101021/cf4a3766/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: roundtrip.png
Type: image/png
Size: 30731 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101021/cf4a3766/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sent.png
Type: image/png
Size: 31114 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101021/cf4a3766/attachment-0001.png>