[IPython-dev] Twisted reactor + IPython0

[Glenn H Tarbox, PhD]

Ville responded to this as well but I'll add some

On Thu, 2008-03-20 at 09:13 +0100, Laurent Dufréchou wrote:
> Hi all,
> 
> Just to drop some more need on threading.
> I've written a little piece of code to create a WX Ipython shell.
> I've used for this a thread with an ipython0 instance and communicate with
> the gui via polling (the first version) and now I'm starting to use
> callback.
> One big need I have and that cannot be resolved easily is to ba able to have
> some sort of Ipython instance with GUI loop support + twisted support.
> The idea:
> 
> Any WXAPP/QTApp   <-- XML RPC --> The twisted loop + Any GUI(QT/WX...) loop
> + Ipython server.

I guess the question is what you're trying to accomplish.  So let me
paste Ville's response which is exactly right and the core of any
response to this kind of issue:

"It really isn't, once we get the twisted reactor to be the "core" of
IPython0. There is lots of fun to be had with this."

So, what is being said here, and what lies at the core of the new /
better / different IPython_x design is an event based architecture
underlying whatever front ends we have.  Embrace giving "the thread"
away....

So, first a claim which should set the stage for a flame war:

>>>>>>>>>>>> Soapbox Warning: <<<<<<<<<<<<<<<<<

Threads are, for virtually all intents and purposes, BAD and should be
avoided like the plague. (never had the plague... maybe its not as bad
as threads)

Of course, the above is wrong, but its way more right than any counter
argument which will be brought forth.

Now, when threads are necessary, they're critical and we can't get by
without them.  But realize, Python has 1 thread... thats right kids,
regardless of the number of cores on your computer, python is only
executing one actual instruction at a time.  Of course, there's a lot of
support the OS provides to your performance experience with another core
such as IO etc... and some libraries do spawn threads underneath... but
python itself is single "true thread"ed which is why you see words like
GIL used and which is another huge issue in the broader python community
and has caused any number of blog / mail group thread wars as python
strives to deal with this multi-core world.

Whats even more important is that threads are almost always unnecessary
and are used, mostly, because people don't know how to accomplish
certain tasks (blocking) without them.  They require resource
synchronization, module segmentation (or other synchronization
migigation strategy) but at least they introduce non-deternimism, lower
performance, and a general impossibility to debugging the intermittent
failures you're sure to experience. (guido, the man himself writes:

"Unfortunately, for most mortals, thread programming is just Too Hard to
get right.... Even in Python -- every time someone gets into serious
thread programming, they send me tons of bug reports, and half of them
are subtle bugs in the Python interpreter, half of them are subtle
problems in their own understanding of the consequences of multiple
threads...."

now lets add that virtually all the gui loops (and this holds true for
most callback style event handling mechanisms) aren't thread safe... so,
you need to go through some kind of prophylactic (always practice safe
threading) when one thread touches something owned by another thread 

<<<<<<<<<<< steps off soap box >>>>>>>>>>>>>>>>>>>

whew, feel better now.

OK, so lets get back to what you were asking... so, what you really want
to do is integrate various event models with pre-existing event handling
mechanisms... 

and away we go....

So, you're using Wx.  Unfortunately, it looks like you're in trouble to
start with although I haven't been able to run this all the ground.
According to some who should know, Wx behaves badly as an event
engine... (apparently stops emitting timers / IO events during modal
windows etc) so the current Twisted Wx Reactor, therefore uses <gulp>
threads to get around this.  This strategy is mostly, why the twisted Wx
reactor doesn't completely work right now... nothing inherently wrong
with the design just some complexity associated with the thread handoffs
which uses a generator and I haven't completely gotten through this one
yet...

the Qt reactor didn't require this as it is well behaved... so I
inverted the design of twisted, kinda... basically, I use Qt as the core
OS integration mechanism and have it call me when certain things need to
happpen... basically that means, simply, that IO which I own and
timing / scheduling is serviced by a few callbacks I register with Qt.
This is mostly what happens with the gtk reactor as well.

So, your sketch using XML-RPC as an integration strategy makes sense if
you're inter-process but, I think, might be overkill in the same
process.  Integrating multiple event loops is nasty because they all
"help" by abstracting (hiding) the underlying OS mechanisms which emit
events to the application layer.  If we can get around this (as I did
with the Qt reactor) we're better off.

Furthermore, I wonder if your strategy will work in general even with
threads... it all depends on what the other event loop does with the
foundational wrapping of the event sources.... so, if you try, for
example, to add your own socket and try and block on select, will that
interfere with Wx?  If you had Wx and Qt in the same process, where
would the window events go?  None of these core engines acknowledge the
existence of any other engines... and would likely behave very badly
with the introduction of such a virus in their process space.

If you're in a different process, all this is an entirely different
discussion and is more about event IO... none of which requires
threading... Qt, Twisted, and Wx all provide very rich mechanisms to
register callbacks for this kind of event handling.

-glenn

> 
> With XML RPC you can just make any action inside Ipython instance.
> I want this to be able to make a WX(and QT) console able to launch any QT/Wx
> windows and thus independent to the Console App.
> 
> With this you can have a really really good dev env for graphical app.
> Moreover designing such a server will be raally good for people who want an
> ipython console, because with this all the wx/QT/... console code will share
> the same ipython object that will save lot of work.
> 
> Glenn, perhaps my idea is stupid, but do you thing we will have a lot of
> performance related problem with this, do you see any issue as you've worked
> a little on something like this??
> Perhaps it is a too big thing for a simple Ipython console...
> 
> Laurent
> 
> -----Message d'origine-----
> De : ipython-dev-bounces at scipy.org [mailto:ipython-dev-bounces at scipy.org] De
> la part de Ville M. Vainio
> Envoyé : jeudi 20 mars 2008 07:20
> À : Glenn H Tarbox, PhD
> Cc : ipython-dev Mailing list
> Objet : Re: [IPython-dev] Twisted reactor + IPython0
> 
> On Thu, Mar 20, 2008 at 7:41 AM, Glenn H Tarbox, PhD <glenn at tarbox.org>
> wrote:
> 
> >  On Thu, 2008-03-20 at 07:33 +0200, Ville M. Vainio wrote:
> >  > On Thu, Mar 20, 2008 at 12:35 AM, Glenn H Tarbox, PhD
> <glenn at tarbox.org> wrote:
> >  >
> >  > > This is where I started originally.  There's a better fix which I
> posted
> >  > >  a while ago.  The below uses polling which is unnecessary.  By
> modifying
> >  > >  shell, you can turn the whole thing into a callback.
> >  >
> >  > The ASPN recipe works like the current GUI backends, so it seems to
> >  > "obviously work".
> >
> >  it works, just spins cycles unnecessarily.  If you look through this
> >  list you'll see the minor changes I made to Shell (posted on Feb 9).  My
> >  code generates a callback when somethings needs to be serviced as
> >  opposed to having a high-frequency polling loop try and determine the
> >  same thing
> 
> Goodie. Frankly, I probably missed the original discussion because I
> thought of it was mostly related to IPython1 (though obviously I can't
> remember why I missed it) ;-)
> 
-- 
Glenn H. Tarbox, PhD

glenn at tarbox.org