[IPython-dev] Fwd: [SciPy-User] ANN: IEP 3.0.beta - the Interactive Editor for Python

[Almar Klein]

Hi IPython developers,

Let me explain about our choice to implement Yoton instead of using pyzmq.


Thanks for the pointer, I saw the announcement of IEP, but not the
> yoton one.  It looks extremely similar to zmq (down to the project
> motto "Easiest. Messaging. Ever", for zeromq a while back I think it
> read "Fastest. Messaging. Ever").


I definitely had ZMQ inspire me for the design. The motto is indeed
shamelessly stolen from zmq's motton; it should not be taken too seriously,
but more as a wink to the zmq project ;)



> But I don't quite see the point of
> reimplementing pyzmq: zeromq is already very good, has fantastic
> python bindings, and there's a very good reason to want your low-level
> networking machinery to be implemented in C++ instead of Python: the
> GIL.
>

The main reason for developing a pure python communication layer was
because I wanted IEP to work with any Python version installed on a system
(Python v2.4 and up), without having to install any extra packages. The
idea is that you can just install the binary version of IEP and get
started.

I already had working code for this (you recalled our discussion from
september 2010) when I learned about zmq and pyzmq. I have seen the
documentation of IPython for the communication between the kernel and
client, and although it all made a lot of sense, I foresaw some
incompatibilities with our ideas about the relation between the kernel and
the client. I cannot recall the details, but I believe the main difference
is that IPython considers the kernel to be "in charge", while IEP considers
the kernel to be a "slave".

Considering that I already had working code, I decided to refactor it to
make the code more clear, and introducing functionality to connect multiple
context in a network: I really like IPython's idea about remote and
collaborative computing. As always, I underestimated the work that was
needed to get it right. In this case I underestimated it *a lot*. But
anyway, this resulted in Yoton.



> Most of ipython's communications code continues to function regardless
> of the load on the python process and whatever may be happening to the
> GIL, and that's a pretty darn important point (there are obviously
> parts that need the gil, but a good amount happens independent of it).
>  Believe, we know: we used Twisted for years, and we saw the impact
> this could have for use cases like engines meant to do heavy-duty
> computing.
>

Indeed, if the kernel is running extension code, the kernel and IDE cannot
communicate. But we can still distinguish between the kernel being dead and
the kernel running extension code. If we connect via localhost, the
socket-pair always remains intact unless the process dies (or the socket is
closed explicitly of course). So if we do not receive heart beat signals,
but we do not detect a socket problem, we know that the process is running
extension code (this is also visualised in the shell-icon in IEP). This
does mean we need a connection via localhost, so we always use a broker
that runs on the same machine as the kernel. This has other advantages; we
also read the kernel's stdout and stderr via a pipe and send these messages
to the IDE via Yoton.

Certainly, zmq has a better performance, but Yoton easily matches the
requirements needed for kernel-ide communication.


But hey, as long as they have fun working on it, that's all that
> matters.  I'm not being sarcastic here, I mean this: people should
> feel free to play even with projects that seem redundant/pointless to
> others, and I'm not criticizing them at all.  *I* don't see the point
> for myself, but that doesn't matter.  And who knows, the ease of
> development of Python may take them in interesting new directions that
> zmq isn't exploring, so in the long run it may prove to be a very
> useful project.
>

Thanks for your understanding :)  Yes we did have a lot of fun. And a lot
of pain and frustration, but we learned a lot in the end.

One more thing that might be of interest. Yoton has a feature for the
pub-sub pattern to slow down the sending side if the receiver cannot keep
up processing messages. Therefore we can guarantee that the user will not
miss any messages. Try running "while True: print(time.time())".

Regards,
  Almar

-- 
Almar Klein, PhD
Science Applied
phone: +31 6 19268652
e-mail: a.klein at science-applied.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120421/7417634e/attachment.html>