[IPython-dev] Curses Frontend

Sun Mar 21 13:28:24 EDT 2010

Wendell,

Fernando and I will be sprinting today and tomorrow on IPython.  We
will get back to you shortly on this stuff.  In the meantime, can you
join the #ipython channel on irc.freenode.net?

Cheers,

Brian

On Sun, Mar 21, 2010 at 7:04 AM, Wendell Smith <wackywendell at gmail.com> wrote:
> Thanks again for your response!
>
> OK, I have very little experience with twisted, and none with 0MQ, and would
> very much appreciate if someone could give me some help there.
>
> My plan so far is:
> 1. Figure out whether to use curses or urwid
> 2. Make the necessary widgets/etc. for curses/urwid, completely independent
> of ipython (mainly, a nice text editing box that can handle coloring, a main
> scrollable text box for the output, and a pop-up window for completions)
> 3. Figure out how to handle coloring - pygments, pycolorize, etc. Ipython's
> built in colorizer is not good enough: firstly, curses doesn't take terminal
> escapes, and secondly, I want to colorize the input as one types, which
> requires handling unfinished input - which the built in lexer can't handle.
> It looks like pygments is the way to go, and I'll need to write a formatter
> for curses (pygments doesn't have one, and curses is a special case, anyway)
> 4. Start working on integration with ipython. While of course ipython will
> be on my mind for the above, I would like the widgets to be sort of their
> own thing, that could be used independently of ipython, for 2 reasons:
> firstly, someone might find them useful anyway, and secondly, it should make
> it easier to do bug fixes and isolate bugs.
>
> I hope to get #1 done today or tomorrow, and get  back to #2. I've already
> gotten a good start on a curses based text-editing, color-capable widget,
> but if urwid looks good, I may drop it and go for theirs.
>
> A couple of questions/problems:
> 1. Curses likes to block while waiting for input - is this ok? Should I try
> and get around this?
> 2. Colorizing. Someone already mentioned that they were thinking of
> switching ipython over to pygments. I could do this - it would make things
> easier for me in the long run. If I don't go that route, then it gets really
> complicated. As coloring the input pretty much requires pygments (built-in
> lexer can't handle unfinished input), there then become several options:
>    a. Require pygments for the curses frontend.
>    b. Don't require pygments, but be completely colorless without it.
>    c. Write some fancy way of using the built in colorizer to handle the
> output, as well as one for pygments for the input. That way, without
> pygments installed, there is still color in the output portion.
> I would, of course, prefer to just convert all ipython to pygments - which I
> think I can do - but of course that's not my decision to make, I'm new here.
> But if that is a good way to go, I'll happily go there.
> Otherwise, I'm leaning towards (b). It's barely more complicated than (a)
> while having advantages over (a), and (c) is just too complicated.
>
> OK, that was a lot. I'm going to get back to looking at urwid, but if anyone
> else has any answers/recommendations/comments/etc., I would love to hear
> them...
>
> -Wendell
>
>
> On 03/12/2010 08:49 PM, Brian Granger wrote:
>>
>> Wendell,
>>
>> I have been busy with other things lately, so sorry I haven't gotten
>> back to you sooner.  I wanted to say a few more things about the
>> overall design of your curses based frontend.  This also applies to
>> other frontends...
>>
>> * For a long time we have wanted to move IPython to a 2 process model
>> that is similar to how Mathematica works.  The idea is to have 1) a
>> simple lightweight frontend GUI/terminal/curses based process that
>> handles user input, prints output, etc and 2) a kernel/engine process
>> that actually executes the code.
>>
>> * The GIL has made this effort very difficult to achieve.  The problem
>> is that if the kernel executes extension code that doesn't release the
>> GIL, the kernel's will appear dead for the networking.  Threads don't
>> help this and Twisted (which we use) doesn't either.
>>
>> * Our solution so far has been to use Twisted and introduce an
>> additional process called the "controller" that manages traffic
>> between the frontend and kernel/engine.  Twisted is amazing in many
>> ways, but it has a number of downsides:
>>
>> - Twisted tends to be an all or nothing thing.  Thus, it has been
>> quite difficult to integrate with other approaches and it is also
>> difficult to *force* everyone to use Twisted.  Also, using Twisted
>> forces us to have a very constrained threading model that is super
>> infelxible.  I can give more details on this if needed.
>> - It looks like Twisted will make the transition to Python 3k *very*
>> slowly (after zope, after everyone drops python 2.x support, etc.).
>> IPython, on the other hand needs to move to Python 3k quickly as so
>> many people use it.
>> - As we have spent more time looking at network protocols, we are more
>> and more convinced that a pure RPC style network approach is really
>> insufficient.  What we really need is more of an asynchronous
>> messaging architecture that supports different messaging topologies.
>> - Twisted is slow for how we are using.  For some of the things we
>> would like to do, we need C/C++ speed networking.
>>
>> So.....for the last while we have been looking at alternatives.
>> Initially we were hopeful about AMQP:
>>
>>
>> http://www.amqp.org/confluence/display/AMQP/Advanced+Message+Queuing+Protocol
>>
>> AMQP looks really nice, but 1) is very complex and heavyweight, 2)
>> adopts a very specific model of messaging that looks to be too limited
>> for what we want.
>>
>> Recently, however, we learned of 0MQ:
>>
>> http://www.zeromq.org/
>>
>> Zeromq has been developed by some of the same folks as AMQP, but with
>> a different emphasis:
>>
>> * As fast as can be (written in C++).  For somethings it is faster
>> than raw TCP sockets!
>> * Super simple API, wire protocol, etc.
>> * Lightweight and easy to install.
>> * All the network IO and message queuing are done in a C++ thread, so
>> it can happen without the GIL being involved.
>>
>> This last point is the most important thing about 0MQ.  It means that
>> a process can run non-GIL releasing extension code and still do all
>> the network IO and message queuing.
>>
>> I have tested out 0MQ quite a bit and have created new Cython based
>> bindings to 0MQ:
>>
>> http://www.zeromq.org/bindings:python
>>
>> Here is a Py0MQ example of what an IPython kernel would look like:
>>
>> http://github.com/ellisonbg/pyzmq/tree/master/examples/kernel/
>>
>> So....my thoughts right now are that this is the direction we are
>> headed.  Thus, I think the model you should use in designing the
>> frontend is this:
>>
>> * Cureses/urwid based frontend that ....
>> * Talks to IPython kernel over...
>> * 0MQ
>>
>> Obviously, you are free to use a 1 process model where Ipython runs in
>> the same process as the curses stuff, but you will run into all the
>> same problems that we have had for years.  I can ellaborate further if
>> you want.  What do you think about this plan?
>>
>> Cheers,
>>
>> Brian
>>
>
>

-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com