[IPython-dev] ipython1 remote exceptions causing local crashes

[Fernando Perez]

Hi Douglas,

On 2/20/07, Douglas Jones <dfj225 at gmail.com> wrote:
> Hi all,
>
> I've been looking at the new ipython1 saw branch and the implications
> of the changes to a project that I'm working on.
>
> I've noticed in my testing that running some code on the engine that
> causes an exception crashes my ipython client. I'm using recent svn
> checkouts of both ipython1(revision 2104) and the ipython shell (which
> lists itself as version IPython 0.7.4.svn.r2010).
>
> Is this a known issue?

Well, it's a feature, not a bug :)  It was actually something that was
deliberately added recently, to ensure that remote exceptions aren't
silently swallowed and that you are forced to know about them.  The
reason is that otherwise swallowing remote exceptions makes for
incredibly hard debugging: your engine threw an exception which went
silently into a log file only, and the client keeps on happily feeding
it new commands to execute, most of which are likely to produce
garbage because some previous quantity you expected to have computed
is not there.

We are trying to bring, within the realm of what's possible, a smooth
transition from a model of single-process synchronous programming to
distributed computing, and exceptions are a particularly thorny issue
here.  We recently spoke to DOE people about  the next-generation
language projects for distributed computing that the DOE/DARPA are
looking into (X10, Fortress and Chapel, see
http://www.ahpcrc.org/conferences/PGAS2006/presentations/Yelick.pdf).
As far as we can tell, no system yet has a completely satisfactory
answer for this, partly because I don't think you /can/ have such an
answer.

We hope we'll offer a model that's reasonable for end users (we are
end users of this, so we're eating our own dog food here), and that
will hopefully balance:

- Feeling like an analytic continuation of the serial programming
model (though remember that you may always find surprises off the real
axis :)

- Offering some utilities that make the common synchronization tasks
needed for regular algorithmic development easy to achieve.

- Exposing enough of the distributed objects to allow more
sophisticated uses. It is still possible that we'll limit access to
parts of the underlying asynchronous machinery for reasons I don't
quite have time to explain right now.  I can go into more detail later
if needed.


But in all of this, feedback from users will be critically important.
We've put a LOT of thought into this, but it's very easy to travel
down a dead end without noticing it, so feel free to speak up.  If we
get this right, this system should be very useful for many people and
for a while, so this is the time to complain if you think we're
barking up the wrong tree on any issue.

> If you need more information, I'd be happy to provide some of the
> traceback information.
>
> The only thing I haven't updated is the version of Twisted installed
> on the machine. Is it necessary to have a newer version of Twisted for
> the ipython1 saw branch? I think Twisted that is installed is version
> 2.4.0.

Yes, saw relies of bugfixes to Twisted that are only part of Twisted
2.5.0.  We do now ship twisted's web2 module internally so you don't
have to track SVN for that.  Basically if you get Twisted 2.5.0
installed (which requires zope interfaces), you should be good to go.

Cheers,

f