[IPython-dev] ctrl+c in gui, event loops, sure i'm missing something...

[Gael Varoquaux]

On Wed, Feb 11, 2009 at 11:05:34PM +0100, Laurent Dufrechou wrote:
> Any clever magic idea welcomed!

This is a recurrent problem that we have been discussing more than once
on this mailing list, although not in these terms.

You are aware of the problems with a single event loop (blocks when doing
calculations) and with threads (fundamentally unreliable and prone to
crashing). If I can rephrase to questions that I can try to answer, I
would say that you have two questions: i) why are we in a situation
different than eg Matlab, or IPython in a terminal? ii) How can we do
better and have something robust, and yet provide reaction to events
(including interruption) when calculations are running.

i) Why are we in a situation different than eg Matlab, or IPython in a
terminal?

With a wxPython frontend for IPython, we are in a very specific
situation: we have an environment for interaction with the user (our GUI)
that is completely embedded and mixed with our execution engine. This is
very powerful, because this means we can couple interaction with what is
being run in the execution engine. 

However, Python is not a language terribly good at isolation. Not only
does it have an incredible amount of global states, but also it has no
thread-safe memory model. Memory coherence across threads in a shared
memory model (ie not an actor-like or message-passing model) is a very
hard problem. AFAIK only Java and .net have solved it, and with a
significant investment in the development of their virtual machine.

We are thus cursed with the fact that our user-interaction code and our
execution engine, when running in the same Python virtual machine cannot
be isolated, and trying to simulate this isolation will lead to quirks,
crashes, or both.

Matlab, or simply running IPython in a terminal, are quite different,
because the interaction with the user is separated from the execution
environment. In this regard, if you want to look at a Python
implementation of this concept, the Sage notebook is a good example, and
Ctrl-C does work quite reliably there.

ii) How can we do better and have something robust, and yet provide
reaction to events (including interruption) when calculations are
running.

I am convinced the solution is going to multiple processes. Actually the
important thing is to have multiple virtual machines between which we
share a	limited amount of states. Going multi-process is a simple way of
doing this. From a pragmatic point of view, you have two option: using
IPython1 to push the execution engine out of process with the IPython1
mechanism, or using the multiprocessing module to keep an thread-like
API, but with several processes.

Brian and Fernando had to fight with a similar problem when exposing the
execution engine over the network: the network-listening code could be
freezed by the execution engine. I seem to remember that they had to use
two processes for an engine: one to run the code, and the other one to
supervise it, but it would be interesting to hear them share there
experience.

My 2 centimes,

Gaël