[IPython-dev] module for blocking on twisted Deferred objects

[Brian Granger]

Barry,

My understanding is that iterate won't block until there is an event
present.  Underneath it simply calls select/poll with a timeout.
Currently, the weakness of this module is that the event loop will
only run while you are waiting (and blocking on) a particular
deferred.  I could put a timeout that allows the user to wait for a
certain time.  The problem is that once that time has passed, the
event loop will stop and twisted won't be able to process the events
when they do come in.  In IPython1, we handle this by having a
block=True/False option on the execute command.

Another way of saying this is the following.  With the blockon.py
module, there are two options:

1.  You are waiting on the result of a deferred using blockon.py and
twisted is iterating through events.

or

2.  Twisted is not processing events and you have interactive control.

To have both interactive control and have the twisted event loop
running you would have to run the twisted reactor in a second thread
and implement some clever way talking between the two threads.  In
this approach you would have to have all twisted code in that second
thread.  My goal with blockon.py was to let the user directly make
twisted calls in the main thread and provide a simple way of iterating
over events.  If you see a nice way of running the reactor in a
different thread, (twisted has a callFromThread method) we could think
about that approach more.



On 12/12/06, Barry Wark <barrywark at gmail.com> wrote:
> Brian,
>
> Does twisted.internet.reactor.iterate block until an event is present
> in the reactors loop? In other words, is the while loop in your
> BlockingDeferred.blockOn polling every TIMEOUT seconds or is it
> blocking until an event is received?
>
> I suspect you're using this to get synchronous-like behavior with
> distributed computations in ipython1. If the computations take hours
> or days, it seems maybe inefficient to poll that much... you might
> want to have an other parameter to the blockOn call that gives the
> timeout. Then the user can decide the polling resolution.?
>
> barry
>
> On 12/11/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > For now I checked it into my sandbox:
> >
> > http://projects.scipy.org/ipython/ipython/browser/ipython/sandbox/bgranger/blockon.py
> >
> > I would love feedback, but will probably work on it more over the next few days.
> >
> > Brian
> >
> > On 12/11/06, Barry Wark <barrywark at gmail.com> wrote:
> > > Brian,
> > >
> > > I at least will be excited to see it, wherever it appears. What about
> > > putting it in ipython1 (a logical place, it seems), plus having a
> > > stand-alone module/package?
> > >
> > > barry
> > >
> > > On 12/11/06, Brian Granger <ellisonbg.net at gmail.com> wrote:
> > > > Hi,
> > > >
> > > > I have written a small python module that provides a synchronous
> > > > interface around the asynchronous calls of Twisted.  This has been
> > > > something that Fernando and I have wanted for a long time, but until
> > > > now I didn't have a solution that was simple and robust.  I could put
> > > > this in ipython1, but it could be useful for many folks so I wonder if
> > > > putting it into ipython/trunk would make more sense.  Thoughts?
> > > >
> > > > Brian
> > > > _______________________________________________
> > > > IPython-dev mailing list
> > > > IPython-dev at scipy.org
> > > > http://projects.scipy.org/mailman/listinfo/ipython-dev
> > > >
> > >
> >
>