[IPython-dev] Message spec draft more fleshed out

Sat Aug 14 12:48:15 EDT 2010

On Fri, Aug 13, 2010 at 8:40 PM, Wendell Smith <wackywendell at gmail.com> wrote:
>  Dear Brian,
>
> Thank you for your answer and your patience... you're really helping me
> understand this, and I appreciate it!

No problem!

> As for curses... I've switched to the urwid library, which, by the way,
> I have already mostly ported to py3k... and the urwid library is set up
> to use any sort of asynchronous main loop you want, with a basic main
> loop written into it, a tornado-based main loop, and a select-based
> mainloop already written, and it's flexible, so one could write a main
> loop on one's own. Input is non-blocking.

That is great - it sounds quite flexible.  One interesting point is
that we are already using a tornado based event loop in the kernel
manager.  There might be some nice ways of integrating things without
the current threaded channels we are using now.

> As for my previous idea, maybe I'm still not understanding, but perhaps
> we could still have a basic system with a kernelmanager object, a
> send_receive function, and two queues, in and out. The send_receive
> function reads messages from the out_queue and sends them, and then
> receives messages from zmq through the ports and then puts them on the
> in_queue, never really looking at what messages are coming in and out,
> just putting them on the queues. The kernel manager object could then be
> set up exactly as I said before, except that it has an additional
> method, process_messages, in which it reads a message from the in_queue,
> determines which method to call, and calls it; the methods do their
> magic, printing, receiving input, whatever, and some put messages on the
> out_queue.

We have definitely thought about doing this type of thing.  However,
we didn't want to make that the only way of handling things because Qt
has the signals/slots thing that work really well across threads.  I
think we should do something like you are proposing in subclassess,
but with one change.  The KernelManager is only responsible for
running the kernel process and creating the channels.  The APIs you
are talking about will be on the channels themselves.  But, you should
subclass the channels, create those queues (the input queues are
already there) and then override call_handlers to simple put the
messages on the out queue.  Then you can implement the callbacks an
process_messages.  It should work quite well and integrate well with
the event loop you end up using.

> As I see it, this sounds great: the queues can be from Queue.Queue, and
> then everything is thread safe, as the send_receive function cold be on
> one thread, the kernelmanager methods on the other, and the two would
> interact only through the thread-safe queues. For an asynchronous
> approach, you have an option to set max_msgs and timeout for both the
> send_receive function and the process_messages method, and call them
> alternately with max_msgs = 1, timeout = 0. This then would make the
> frontend programmer's job easy: all they need to do is get their main
> loop to frequently call send_receive and kernelmanager.process_messages,
> with max_num and timeout set appropriately, and then fill in the other
> methods from the kernel manager to provide output and input.
>
> This is a bit simplified... we might need prioritized queues, or
> send_receive might need to add tags to the message to say which channel
> they came in on... but that shouldn't be difficult, if the rest of it
> makes sense.
>
> Would this work? If not, what am I missing?

I think this will work.  I would give it a shot and that will show it
you need to tweak the design.

The only caveat that people have run into is that it can be quite
subtle to synchronize the messages on the different channels.  That is
why the SUB channel has a flush method.  But I would still just go for
it and see how it goes.

Cheers,

Brian

> Thank you again for your patience,
> Wendell
>
>
> On 08/13/2010 07:16 PM, Brian Granger wrote:
>> Wendell,
>>
>> On Fri, Aug 13, 2010 at 3:04 PM, Wendell Smith<wackywendell at gmail.com>  wrote:
>>>   Hello all,
>>>
>>> I have been looking all these documents over, and wondering if perhaps
>>> we could have some object (descended from KernelManager) that would be
>>> constructed to perfectly match the message spec, such that any message
>>> received would translate to a functional call (i.e.
>>> MessageManager.execute(self, header, code, silent=False)), to make it
>>> easy for someone to simply write an object that descends from
>>> MessageManager (or whatever we call it) and fill in the methods. This
>>> would also then serve as a message spec - it should be created such that
>>> it can receive any valid message and sends only (and can send all) valid
>>> messages.
>> The KernelManager classes and ZMQChannel classes are about as close as
>> we can get for now.  There are a couple of different issues:
>>
>> 1. The handler methods that you are talking about need to be called in
>> the main thread.  But all of the channels are receiving msgs in a
>> second thread.  The call_handlers method needs to be overridden in a
>> way that causes the true handler methods to be called in the other
>> thread.  Each toolkit has its own way of calling functions in other
>> threads, so that is difficult to do in a general way.  Furthermore, in
>> a terminal where there is no event loop, there really isn't a way of
>> calling a method in a different thread.  Thus, the yet-to-be-written
>> subclasses of KernelManager and the channels will have to simply put
>> the received message in a Queue and the main terminal thread will have
>> to poll that queue.  But an important question is this:  does curses
>> have a way of calling functions in the main thread?  If not, we will
>> have to develop a custom KernelManager and channel classes that use
>> Queues and polling.  I have started some of this in
>> blockingkernelmanager.
>>
>> 2. We are moving towards a model where "The message is is API"  Thus,
>> we don't want to hide the actual messages from frontend code.  You
>> really need all of that information and because of (1), we can't
>> really easily put it into handler methods.
>>
>> 3.  We still don't quite know what is needed by different frontends,
>> so it is difficult to identify the common code yet.  As time goes on,
>> we may realize all frontends use similar logic and we can abstract
>> that out properly at the time.  But for now we are in the
>> wild-wild-west.
>>
>> 4.  Everything is truly asynchronous.  It takes a while to get used to
>> this fact as it is *very* different than the old terminal based
>> IPython.  Because each frontend will handle that asynchronicity in
>> different ways, it is very difficult to come up with abstractions
>> beyond the messages that are universal.
>>
>>> Of course, this may not make sense, and I may not know what I'm talking
>>> about - I don't know much about the zmq communication, and was sort of
>>> hoping to stay focused on the fancy console frontend without delving too
>>> deeply into that, but if others agree with me but no one with better
>>> knowledge wants to do it, I would be happy to write the necessary code
>>> myself, but again, I'm probably not the one best able to do it.
>>>
>>> Speaking of combined code, it would also be nice to have a
>>> frontend.pygmentize module that covers pygments coloring for input,
>>> output, prompts, and tracebacks, providing a lexer and a style from
>>> config, (formatters would depend on the frontend), and also perhaps some
>>> object that takes a formatter and provides all these tools for the
>>> frontend, perhaps even descending from KernelManager and just having
>>> methods that manage these. That would be nice. I could work on that too,
>>> and would be happy too - I just noticed that at least Evan and I have
>>> written pygments code, and it would be nice to avoid code duplication.
>> I agree, and if you want to take what Evan has done and create a
>> common base class
>> that all frontends can use and appropriate frontend subclasses for qt,
>> curse, terminal
>> that would be great.  Not sure it should (at least yet) be descending
>> from KernelManager though.
>>
>>> Anyways, I just feel like we've got 4 people working away on 4 frontends
>>> without too much communication going on about useful common code, and it
>>> would be nice to get that sort of work delegated out before we all go
>>> and write our own versions of the same tools.
>> Part of the challenge we are having is that there is so much code
>> being written currently that none of us can keep up with it all.  Our
>> current model is that my kernelmanager branch is the "common code
>> base" that all 4 frontends are using.  There is definitely repeated
>> code in the various frontends and over time we will move that into the
>> common base.  But you are stepping in very early in the process,
>> before all the APIs are very solid.  But part of what we want and need
>> is for the various frontend developers to give it a shot and let us
>> know what things they need in the common base.  But I think the way
>> that needs to go is that each frontend does it on their own first, to
>> see what works best for them and then we try to identify the
>> commonalities.
>>
>>> Please let me know if this makes sense and is a good idea - I am
>>> certainly not the most knowledgeable here, and if I seem to be missing
>>> something, please let me know!
>> No, you have definitely noticed the most important point - this stuff
>> is not simple and it is definitely not done!  But please let us know
>> if you have questions.
>>
>> Cheers,
>>
>> Brian
>>
>>> -Wendell
>>>
>>> On 08/13/2010 04:27 PM, Brian Granger wrote:
>>>> I have this in a browser tab and will review it soon.
>>>>
>>>> Cheers,
>>>>
>>>> Brian
>>>>
>>>> On Thu, Aug 12, 2010 at 1:13 AM, Fernando Perez<fperez.net at gmail.com>    wrote:
>>>>> On Wed, Aug 11, 2010 at 2:50 PM, Brian Granger<ellisonbg at gmail.com>    wrote:
>>>>>> Very good points.  I think we should just copy this description into the
>>>>>> message spec.
>>>>> I just updated the doc and pushed to trunk and a build with Min's text:
>>>>>
>>>>> http://ipython.scipy.org/doc/nightly/html/development/messaging.html#heartbeat-for-kernels
>>>>>
>>>>> Modulo final feedback, that design spec is reasonably complete, as far
>>>>> as I'm concerned.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> f
>>>>>
>>>>
>>> _______________________________________________
>>> IPython-dev mailing list
>>> IPython-dev at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/ipython-dev
>>>
>>
>>
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>

-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com