[Baypiggies] [ANN:Py4Science at Cal] October 27: Parallel IPython with ZeroMQ

[Fernando Perez]

Hi folks,

for next week, we have Min Ragan-Kelley presenting the new machinery
for parallel work in IPython that is now based on the ZeroMQ
architecture we discussed a few weeks ago.

Please forward this to any interested colleagues.

Cheers,

f

*Py4Science at Cal meeting*

Title: Parallel IPython with ZeroMQ

Speaker: Min Ragan-Kelley, U.C. Berkeley

Abstract:
IPython has provided interactive parallel computing for some time, but
there are limitations to the current use of the Twisted library for
networking.  Twisted is a high-level utility with many benefits, but
makes some critical functionality, such as non-copying sends,
impossible.  Using the low-level message passing library ZeroMQ for
communication has given us many advantages over Twisted, but also
introduces new challenges.  It is now possible to send a string or
numpy array (or anything that provides the buffer interface) with
exactly zero in-memory copies, dramatically improving performance for
large data messages.  In addition, a new function-based interface
mirroring multiprocessing's 'apply(f,args,kwargs)' allows executing
remote code to be more intuitive, so there will no longer be any need
to package code into strings for remote execution.  Preliminary tests
show 40-100x performance improvements over Twisted code for
high-frequency job submissions.