[IPython-dev] ipython1 and synchronous printing of stdout

[Barry Wark]

On Jul 22, 2008, at 3:25 PM, "Brian Granger" <ellisonbg.net at gmail.com>  
wrote:

>> As always, I think working code is better than good theory, so please
>> read the following with that in mind. I think that the opinion you
>> express below may put us on the road to an architectural mistake in
>> the redesign of ipython. My understanding of the historical context  
>> is
>> that the current ipython0 interpreter is direclty tied to the  
>> frontend
>> in a way that makes new frontends or stand-alone interpreters
>> difficult. The solution, as embodied by the the ipython1 codebase and
>> the evolving IPython.frontend package is to establish a separation
>> between interpreter and frontend to prevent the type of dependency
>> that exists in the current code base.
>
> One thing that is absolutely critical is loose coupling.  All the
> parts of IPython need to depend only very weakly on each other, and
> interact through clean, standardized interfaces (I don't mean this
> word in the script zope.interface sense though).  The big problem with
> the traditional IPython codebase is that everything is welded
> together.  The question is what types of design patterns will make
> this possible.
>
> From this thread, I see a few options:
>
> 1.  Traits.  While I really like the Traits model and I think it is
> highly appropriate for GUI programming, I think it would lead to too
> tightly coupled code.  Also, we can't use Traits in the core of
> ipython because it has compiled code.
>
> 2.  Plain callbacks.  At least a couple of us seem to think that an
> interface that has lots of callbacks becomes very cumbersome.
>
> 3.  The Observer/Delegate patterns as are typical is Cocoa.  I know
> that Gael is not fund of this, but I do think this pattern is much
> better than plain callbacks, as it give a nice formal structure to
> everything.  Also, I agree with Barry that this pattern is pretty much
> what you get when you try to do plain callbacks properly.
>
> Thus, my own conclusion is that the Observer/Delegate pattern is
> probably the best option for us and will lead to the loose coupling
> that we need.
>
> Barry, what would it take to put together an example of this pattern?
> Will we need some amount of infrastructure to make this possible?

I'd be happy to write up an example and a draft implementation.  I'm  
going to be mostly offline, traveling, until the weekend. I'm most  
familiar with the Cocoa implementation so I'd like to do a bit of  
research of alternatives before I write the spec. Therefore next week  
is the realistic timeline.

Off the top of my head, I think a synchronous notification center  
implementation in I.k.core and an addition to the engineservice API  
along with an asynchronous notification center impl in I.kernel would  
cover the use cases. I will write up a functional/technical spec and  
solicit feedback before I make any decisions though.

Besides the notificatipn center implementation(s), I don't think any  
more infrastructure is required beyond modifying the interpreter to  
actually fire the appropriate notifications (support for obvious  
callbacks such as Gael's use cases should be easy; the notifying  
worskspace ala Robert's work will obviously require more effort). As  
you pointed out, this is really just carefully implemented callbacks  
with a loose coupling.

I'm glad we're dealing with this issue early. I think getting it right  
will pay big dividends in the future.

Cheers,
Barry

>
>
> Cheers,
>
> Brian
>
>> Now, back to the issue at hand. In developing frontends using the new
>> architecture, you have noticed that there are a lot of things that  
>> the
>> new system is missing. In general, these fall into the category of  
>> the
>> frontend wanting to monitor and/or modify execution in the  
>> interpreter
>> at a more fine-grained level than entire code blocks (the only option
>> currently). There is a standard pattern for handling this type of
>> notification, the Observer pattern, and this type of fine-grained
>> modification, the Delegate pattern. Propper implementation of either
>> produces minimal overhead (and it's worth remembering that by writing
>> IPython in python instead of a lower level language, we've all  
>> already
>> stated our preference for abstraction over raw efficiency) [1]. The
>> alternative, that I believe you are proposing below and in the  
>> current
>> thread on adding a callback to the exception handling code in the
>> interpreter, is to add callback hooks to the interpreter so that one
>> (or more?) callbacks can be attached to specific events in the
>> interpreter's execution flow.
>>
>> In fact, I think these are not really different solutions, just
>> different implementations of a similar idea. You propose handling
>> observer notification from within the shell, whereas I propose
>> handling it in an intermediary entity (the notification center). The
>> callback approach is more explicit -- the interpreter must explicitly
>> expose a callback hook and the frontend must explicitly register  
>> *with
>> the interpreter* for that callback -- but less general. Unless we
>> create a callback registry etc. in the shell, essentially duplicating
>> the notification center idea, then when we want to add a new
>> notification (e.g. for exceptions as in the currently running  
>> thread),
>> we will need to duplicate the callback calling code in the shell
>> rather than just marking that event for notification by the
>> notification center. The observer/delegate combination allows the
>> interpreter to remain agnostic wrt the interface of the frontend and
>> vice-versa (and huge win, in my opinion). By adding an intermediary  
>> --
>> the notification center and/or delegate dispatcher -- we avoid the
>> issue of frontends and interpreters being closely tied by contract  
>> and
>> make it easier to provide the same interface directly (between
>> frontend and shell as in your Wx frontend) or locally or remotely via
>> a twisted service (as between and engine and the frontend in my Cocoa
>> frontend).
>>
>> I know that you're on a deadline and need to write code, but I would
>> urge a little bit of restraint and planning before you commit to the
>> callback-only approach. I think avoiding this type of dependency
>> between shell and frontend will benefit us in the long run.
>>
>> Barry
>>
>> [1] For example, firing a notification when no observers are
>> registered for that notification should not incur significant  
>> overhead
>> beyond a couple of method calls, and perhaps a dictionary lookup,  
>> even
>> in a naive implementation.
>>
>> On Sun, Jul 20, 2008 at 9:50 PM, Gael Varoquaux
>> <gael.varoquaux at normalesup.org> wrote:
>>> On Sun, Jul 20, 2008 at 09:36:43PM -0700, Barry Wark wrote:
>>>> There are, I think, several "events" for which a frontend might  
>>>> want a
>>>> callback. Opening files,
>>>
>>> Hum, in what sens? What would trigger this event?
>>>
>>>> stdout/stderr output,
>>>
>>> Sure, this is why I came up with SyncOutputTrap, that allows me to  
>>> pass a
>>> callback to write. I add a wx.Yield() in this calback and my UI can
>>> refresh each time you write something to stdout/stdin.
>>>
>>>> and modifications to the user_ns are some that come immediately to
>>>> mind. Opening files might be of interest if the frontend wants to
>>>> notify the user that there is file output to be retrieved from the
>>>> engine, stdout/stderr for the resason Gael outlines, and  
>>>> modifications
>>>> to the user_ns to allow the frontend to keep an up-to-date view  
>>>> of the
>>>> user_ns (ala Matlab's Workspace browser) without querying the the
>>>> entire engine user_ns after every executed block.
>>>
>>> Hum, I am not too excited about firing events when the namespace  
>>> gets
>>> updated. This could slow things down a lot. I general I am not too
>>> excited about having events fired all over the place. This can  
>>> lead to
>>> tedious code with getters and setters and a udge performance loss.  
>>> I am
>>> much more interested in using objects with a standard interface  
>>> (such as
>>> a file, for instance) and then, if you want getters and setters, or
>>> the possibility to add callabcks, you do it in these objects.
>>>
>>>> Apple's NSDistributedNotificationCenter API
>>>> (http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSDistributedNotificationCenter_Class/Reference/Reference.html 
>>>> )
>>>> seems like a good pattern for this task (I'm sure there are  
>>>> equivalent
>>>> APIs in other frameworks). Basically, the frontend can register for
>>>> notifications by name. The interpreter then fires notifications  
>>>> for all
>>>> events and they are passed on, by the notification center, to all
>>>> registered observers.
>>>
>>> Well, I don't really like this kind of code. I find it clunky, and  
>>> you
>>> end up have notification registrations all over the place (reminds  
>>> me of
>>> wx). I actually prefer a lot the traits model, or the VTK model,  
>>> AFAIK,
>>> where objects can be there own observers. As a result you do not  
>>> have
>>> notifications blending in objects where they do not belong. For  
>>> the core
>>> frontends, obviously, we do not want to add a dependence to  
>>> traits, but I
>>> must imagine I cannot imagine going much further without it. To  
>>> take your
>>> example of updating the namespace view, all you have to do is to  
>>> pass a
>>> traited dict, instead of a dict, to the interpreter as a  
>>> namespace. Then
>>> you can have your namespace update automatically (AFAIK Robert is
>>> actually working on a branch where he does that).
>>>
>>> If we start adding callbacks whereever we need them, I am worried  
>>> we'll
>>> end up with callbacks everywhere. I think we need to add callback  
>>> only
>>> where they are absolutely necessary.
>>> My two cents,
>>>
>>> Gaël
>>>
>> _______________________________________________
>> IPython-dev mailing list
>> IPython-dev at scipy.org
>> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>>