[Async-sig] asyncio <-> twisted interoperability

Wed Jun 8 17:15:10 EDT 2016

Similarly, txtulip has existed for some time.  It would definitely be more performant and simpler if we could skip the subclass check, but we don't need this to happen in this cycle.

-glyph

> On Jun 8, 2016, at 11:17 AM, Ben Darnell <ben at bendarnell.com> wrote:
> 
> On Wed, Jun 8, 2016 at 1:52 PM, Yury Selivanov <yselivanov at gmail.com <mailto:yselivanov at gmail.com>> wrote:
> Glyph, Ben, Amber,
> 
> So what’s the resolution on Future.__isfuture__ and fixing the
> isinstance(obj, Future) checks from asyncio?
> 
> 3.5.2 RC is only few days away, I can still make the change if
> it’s a blocker for Twisted and Tornado.
> 
> None of this is blocking Tornado - we shipped asyncio integration six months ago. There's some room for improvement, but I don't think there's a clear enough mandate to squeeze something in for this release. I'd rather take the time to sort out a more complete plan before the next release. 
> 
> -Ben
>  
> 
> Yury
> 
> > On Jun 6, 2016, at 5:35 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> >
> >>
> >> On Jun 6, 2016, at 14:21, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote:
> >>
> >> On Mon, Jun 6, 2016 at 1:54 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> >>
> >>
> >>> On Jun 6, 2016, at 08:29, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote:
> >>>
> >>> On Sun, Jun 5, 2016 at 10:16 PM, Glyph <glyph at twistedmatrix.com <mailto:glyph at twistedmatrix.com>> wrote:
> >>>>
> >>>> On Jun 4, 2016, at 13:25, Ben Darnell <ben at bendarnell.com <mailto:ben at bendarnell.com>> wrote:
> >>>>
> >>>> If things are so sensitive to minor changes in timing, doesn't that set the
> >>>> bar impossibly high for interoperability?
> >>>>
> >>>>
> >>>> The sensitivity is not to changes in timing - i.e. when the wall-clock runs,
> >>>> or ordering of non-deterministically ordered events - but rather to
> >>>> reentrancy - whether certain things complete synchronously while the caller
> >>>> is still on the stack and can depend on them having done so upon return.
> >>>>
> >>>> The recommended way of writing tests within Twisted these days depends
> >>>> heavily on `.callback´ synchronously resolving a Deferred, which is what
> >>>> adding a call_soon breaks.
> >>>
> >>> That's interesting, and also potentially worrisome (for interop, I'm
> >>> not saying Twisted is wrong here).
> >>>
> >>> I think asyncio depends on the opposite: that if you add a callback to
> >>> a Future that's ready it does *not* immediately run. Asyncio's promise
> >>> is pretty strongly that callbacks are serialized (no callbacks running
> >>> inside other callbacks). IIRC we experimented with other semantics and
> >>> found that it was harder to reason about. (IMO if you *know* a Future
> >>> is ready why add a callback to it rather than just calling the damn
> >>> thing if that's what you want?)
> >>
> >> I don't think Twisted is necessarily right here either.  You're absolutely right that it's easier to reason about reentrancy in some cases if you have a might-be-fired-might-not Future vs. the same sort of Deferred.  I like the property where you can do:
> >>
> >> def test(test_case):
> >>     a = asynchronously_something()
> >>     test_case.this_must_not_have_a_result(a)
> >>     cause_a_to_fire()
> >>     test_case.assertEqual(test_case.this_must_have_a_result(a), something)
> >>
> >> but this is (somewhat) opposed to the fact that call_soon means you never get the nasty surprise where the callback added in the middle of a function gets run before the rest of it does.  So I think there are good properties in both cases and given some thought it is probably possible to map between them, but Deferred is lower-level here in the sense that it provides a way to do this both with the event loop and without.  You can always call_soon(deferred.callback) but you can't go the other way and force a Future to resolve synchronously - right?
> >>
> >> Right. I'm still unclear on what the compelling use case for that is (other than that Twisted has always done this). Is it performance? Is it callback ordering?
> >
> > Very, very early in the development of Deferreds, they worked the way Futures do; we changed it mainly to reduce coupling to the event loop so that we could test general-purpose algorithms (like gatherResults) without needing to spin an event loop to do it.  So the main use-case is testing.
> >
> >> I suppose Deferred has a method to mark it done.
> >
> > Yep; ".callback".
> >
> >> Does that immediately run the callbacks?
> >
> > It runs callbacks up to the point that the first one returns a Deferred, and then it waits for that one to be fired to continue running the chain.
> >
> > There is an exception here, where Deferred effectively opts in to Future-like behavior in a very specific case: if you are recursively giving results to a Deferred X that would un-block a Deferred Y inside a callback on Y, Y will not execute its own callbacks reentrantly; it waits until the current callback is done.
> >
> > So while the semantics of .callback() on a Deferred are clear-cut with respect to that Deferred itself, "continue any Deferreds waiting upon it" is a callback-like structure that is slightly squirrely in a very call_soon-like way to avoid surprise reentrancy and RecursionError explosions when having a structure like an asynchronous 'for' loop.
> >
> >> Or does it come in two flavors? Can a Deferred that's marked done ever revert back to being not done?
> >
> > No.  A Deferred that has been called back stays called back; callbacking it again is always an error.  However, it may pause running its chain if you return another Deferred in the middle someplace; the way to resume it is to give the inner Deferred a result; the outer one cannot be otherwise compelled to continue.
> >
> >> (I believe I once read the Deferred code enough to be able to find the answers, but I'm afraid I've never really needed what I learned then, so I've forgotten...)
> >
> > Happy to fill in these blanks; they're (mostly, modulo the weird exception for callbacks-in-callbacks) straightforward :).
> >
> > -glyph
> >
> > _______________________________________________
> > Async-sig mailing list
> > Async-sig at python.org <mailto:Async-sig at python.org>
> > https://mail.python.org/mailman/listinfo/async-sig <https://mail.python.org/mailman/listinfo/async-sig>
> > Code of Conduct: https://www.python.org/psf/codeofconduct/ <https://www.python.org/psf/codeofconduct/>
> 
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org <mailto:Async-sig at python.org>
> https://mail.python.org/mailman/listinfo/async-sig <https://mail.python.org/mailman/listinfo/async-sig>
> Code of Conduct: https://www.python.org/psf/codeofconduct/ <https://www.python.org/psf/codeofconduct/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160608/8cafeea4/attachment.html>