[IPython-dev] Update on IPython recent developments

[Brian Granger]

Hello all,

The purpose of this email is to update IPython developers on the state
of IPython1, which is currently focused on bringing interactive
parallel and distributed capabilities to IPython.  Over the past month
or so, we have done a good amount of refactoring of IPython1.  As of
now, the current development branch for IPython1 is "saw" rather than
"chainsaw."  Because people are still using chainsaw, we will maintain
it for a while (maybe another month), but the saw branch has a lot of
new things.  I should say, that we are not yet recommending that
averge users start to use "saw" yet.  There are a few more things we
need to do before it is a full replacement of the "chainsaw" branch.
But anyone following the development of IPython1 closely, definitely
should look at "saw."

Below is a summary of the new things in the saw branch.  For an
overview of our overall design, checkout the following wiki pages:

http://ipython.scipy.org/moin/IPython1
http://ipython.scipy.org/moin/Parallel_Computing

Design Changes
=============

The core design elements are documented in the engine and controller
interfaces.  These can be found here:

For the engine:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/engineservice.py

For the controller:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/controllerservice.py

The big different in the saw branch is that the controller interface
has been simplified dramatically.
The controller is now basically something to which engines connect.
How those engines
are presented to clients is no longer specified in the controller interface.

The reason we have done this is to make is easy for developers to
quickly develop (using)
adapters new high level client interfaces to a a controller (a set fo
engines).  The client interface
that was in the chainsaw branch is now called the multiengine
interface and is documented here:

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/multiengine.py

This controller interface is exposed to various network protocols in
the modules:

multienginepb.py
multienginevanilla.py

It should now be very easy for people to use the multiengine case as
an example in developing new controller interfaces.  Some of the
interfaces that we are thinking about:

A task farming interface
A map/reduce style interface

The nice thing about this new design is that by using adaption a
single controller instance can be adapted to multiple interfaces and
each of those interfaces can be exposed to the network
in multiple network protocols.

Please feel free to email us if you have questions about these things.

Testing Facilities
==============

We have been working hard to get our tests up to speed.  There have
been two big developments on this front lately:

* New testing utilities.  Fernando has done a great job in adding new
testing utilities that allow lineline doctests, standalone doctests
and standard unittests to be integrated together in a seamless
fashion.  These utilities also allow IPython code to be tested in
doctests :)  Have a look here for these new utilities, examples and
templates.

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/test

* Much better test coverage.  With the refactoring we have done, we
also have improved our test coverage a lot.  There is more to do, but
we are beginning to have a good deal of confidence in the core parts
of IPython1.

Error Propagation
==============

We are now able to propagate most types of exceptions over the
network.  While some of this was being done before, the new
implementation is much better:

* The exception types are preserved as the hop along the network.
* Exceptions are raised in the users namespace
* Errors are now reported in an agressive manner.  This means that any
error is immediately sent back to the user
the instant is happens.

Client Changes
============

The old client that is documented on our wiki pages is still
supported, but will be depreciated soon.
We are working on a new version that will use a better network
protocol.  This new version will have the same exact API so any
existing code that uses that client will still work.

We have a new client based on Twistsed's Perspective Broker protocol.
This client is fully asynchronous and in combination with our new
blockon.py module, code can be written in a nice sort of hybrod
synchronous/asynchronous manner.  We still don't have documenation of
this, but that should emerge soon.  Have a look at

http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/blockon.py
http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipython1/kernel/multienginepb.py

If you are interested in this client.

When the "saw" branch in ready to replace "chainsaw" formally  we will
let people know on the dev and users lists.

Cheers,

Brian