[IPython-dev] ipython1 and synchronous printing of stdout

[Barry Wark]

Taking a bit of my own medicine—that working code is better than
awesome theory—I've checked in the ipython-notification branch and
proposed it for merging into trunk. It includes a functional spec in
/blueprints (it's a bit tounge-in-cheek; I hope no one is offended) as
well as an implementation of the NotificationCenter I've been
discussin in IPython.kernel.core.notification.

Hopefully code will answer more questions than the "fog" we've created
so far. I've added a couple of comments below, but I'd guess a quick
look at the code is more valuable.

B

On Thu, Jul 24, 2008 at 10:45 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> I realize that Barry and Brian will be going offline, and I'll be
> traveling tomorrow back home as well, so this is mostly for the
> conversation to continue later, but while the ideas are in my head.
>
> I greatly appreciate that everyone is trying to wrap their heads
> around a rather abstract and difficult question, but I do worry a bit
> that it's getting too far for me to make a solid judgment on which
> road will really work better.  We all agree on the basics we want
> (loose coupling, acceptable performance, etc), but we're trying to
> figure out a flexible architecture that lets us implement things we
> need:
>
> - across gui toolkits
>
> - out of process (which more or less implies also in a
> network-independent way, since something that can be communicated out
> of process can be sent over the network,  given that we're not using
> shared memory tricks).
>
> The question is how to get from here to there with the least amount of
> pain and keeping us all one happy family :)  I'd really prefer it if
> we didn't end up with a fork just because the Wx/traits frontend gets
> bogged down in a heavy and over-abstracted notification API, so I hope
> we can figure out a viable solution for the lot.

I hope you'll see that this is a pretty light-weight implementation...

>
>
> On Thu, Jul 24, 2008 at 1:23 PM, Barry Wark <barrywark at gmail.com> wrote:
>
>> As Brian points out below, delegation can lead to some unexpected
>> dependencies. Let me give one other example: suppose we use subclasses
>> of parameterized objects to add behavior to the interpreter.
> [...snip careful description of inheritance issues...]
>
> Note that the problems you point out here are simply the generic
> issues that exist with subclassing with Python: any time you use a
> subclass in python, you have to worry about the parent class in the
> future growing new attribute names that may collide with your own.
> That's actually a flaw of the Python object model that is rarely
> discussed, and which languages like C++ in fact do not have.
>
> But this particular issue is a general OO/Python one, that has nothing
> to do with a notification pattern or event handling.

I wasn't trying to suggest that this is an observer pattern-specific
problem, just that this particular problem is likely to crop up in the
notification/callback context if we're not careful. Probably not a
great reason to go with one solution over the other, however.

>
> [Brian]
>
>>> The real benefit of the observer/notification model is that everything
>>> can be completely transient.  For example, you might have a notifier
>>> that observes the user's namespace.  But you might only want it to
>>> operate when the user opens a specific window.  Another example is
>>> when an observer is hooked up to an unreliable network connection.  So
>>> I guess I would ad another design constraint that I have until now
>>> kept in the back of my mind:  we need loose coupling that is also
>>> dynamic and transient.  And for these situations, I still think the
>>> observer pattern is the best solution.
>>
>> Just to reiterate and paraphrase Brian, the observer pattern lets the
>> Interpreter not know *anything* about the frontend (even whether it
>> exists or not) that is observing its behavior and it lets the frontend
>> not know *anything* about the Interpreter's implementation except that
>> it will fire notifications for the events defined in its interface.
>> So, when a new event is added to the Interpreter, no observer code
>> needs to be modified (it doesn't even need to know that the new
>> notification is fired). If the Interpreter's implementation is changes
>> so that, e.g. notification of writing to stdout happens immediately
>> after writing or after some short delay, none of the observing code
>> needs to be reviewed as long as the original Interpreter API didn't
>> specify the exact time. In other words, using an intermediary
>> notification center enforces a loose coupling; the only dependency
>> between Interpreter and frontend is the API that defines the
>> event/notificaiton itself.
>
> The part that worries me is building YAEHA (Yet Another Event Handling
> API)... Matplotlib did exactly that, and while it does allow you to
> wire events using pure matplotlib code, it adds real, non-trivial
> complexity to the code and makes it harder to fit into a specific
> event handling API 'from the front'.  All GUI systems have their own
> event models, so does Twisted, and I'm really not convinced that
> IPython should grow its own (next to matplotlib's, and Traits', and
> WX's, ...)

I appologize if I gave the impression I was trying to create YAEHA.
This is about non-frontend events. There shouldn't be much, if any
overlap between GUI events and the notifications we're talking about.
It really is just a clean implementation of callbacks, nothing more,
hopefully not too much less :)

>
> On the other hand, I think this is a discussion that is beginning to
> get far enough into design abstractions that it's escaping my
> (admittedly very limited) ability to gauge fruitfully the real world
> outcomes.  So here's a concrete proposal:  Gael is already hard at
> work (and far along) on a WX frontend that Enthought is funding for
> use in Envisage apps.  Since this is real, working code, let's see
> where it goes.  In the meantime, Barry can play with the design for a
> lightweight notification center idea, and actual implementation will
> likely be very enlightening.  I've got my hands more than full trying
> to get at least a first cut of all the tests that appeared (once I
> completed the testing plugin, all sorts of little things got picked up
> as tests that now need to actually work).  Since we're trying to push
> a release before scipy, this is really more than we can do all at the
> same time.
>
> With Gael's and Barry's implementation on hand,  I think it will be
> easier to see if the notification center can be made lightweight
> enough for adapting it to Traits/Twisted/Cocoa/whatever without undue
> pain,  or if an object-by-object approach ends up being better.  I can
> easily see it being both: object parametrization for certain kinds of
> architectural customization with a little message center for
> lightweight notifications.

As always, the best solution is likely the middle road between two good options.

>
> But honestly, I'm finding it hard to see far enough into the fog with
> pure 'whiteboard' thinking.  We always brag about how nice it is to
> use python to actually implement ideas rather than only think about
> them in the abstract, I think in this case this may be the wisest
> approach.
>
>
>> [1] Re: performance
>> I believe that Fernando's "no-op" parameter must incur *some*
>> overhead; either the Interpreter must test for None, or the
>> Interpreter must call a method in Fernando's object which does
>> nothing. In other words, there's no free lunch. I can't think of a way
>> to add the ability to notify observers and/or modify the behavior of
>> Interpreter via delegates with truely zero overhead while avoiding the
>> parallel implementation problem that I describe above.
>
> No, the None check is only done at isntantiation time of the main
> objects.  Afterwards, it's the responsibility of the  subclasses to
> produce/manage events for specific behavior.  The default object will
> for example have a .write() method that simply prints, while an event
> handling subclass would have its .write() be Traited so that a GUI
> could listen in, or a Twisted one would handle it over the network.
> So it really is zero-overhead, simply because in the default, there is
> no event production/consumption at all.

Ah, I see. You're proposing moving _all_ the event generating code
into the parametrized objects. Now I get it. I've written into the
spec O(1) performance of my implementation, but it currently depends
on the (undocumented) complexity of the builtin set type. We'll have
to see if performance is acceptable.

>
> So from a performance perspective, that approach really produces
> exactly zero cost.  The question that I think will be best answered
> with a bit of prototyping, is whether the architectural costs become
> greater than those of a message center.

And I suppose whether the added number of objects and method calls
ends up being the same cost as the notification center implementation.

>
> Gael just emphasized that in the code he's doing,  if it turns out the
> message center approach works well, there's little cost in adding it
> later on.  So it's really not an either/or issue, just one of getting
> functioning prototypes  working  quickly so we can judge based on the
> actual code rather than pure abstractions.
>
> Does this sound like a reasonable plan?

Absolutely. I look forward to hearing you (and others') opinions on
the ipython-notification branch.

Cheers,

Barry