[Async-sig] asyncio <-> twisted interoperability

[Ben Darnell]

On Fri, Jun 3, 2016 at 10:17 PM, Yury Selivanov <yselivanov at gmail.com>
wrote:

> [..]
> > Actually, +1 for Ben’s suggestion to revisit if asyncio.Futures really
> need call_soon scheduling of callbacks. Specifically, a torrent of
> call_soon doesn’t really achieve “fair scheduling” as suggested by Yury
> anyway:
> >
> > https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445
>
> One quick story about call_soon: while working on uvloop I once spent a
> couple of hours debugging why asyncio/sslproto.py didn’t work.  Turned out
> that SSLProtocol, in its connection_made scheduled some callbacks with
> loop.call_soon; and data_received did that too.  Turned out that libuv
> called data_received one loop iteration earlier, which resulted in a race
> condition.  What I’m trying to say here, is that making a subtle difference
> in how callbacks are scheduled can introduce some hard to debug bugs.
>

If things are so sensitive to minor changes in timing, doesn't that set the
bar impossibly high for interoperability? In Tornado, we have similar
concerns around "fair scheduling" of coroutines, but we solved this by
using our equivalent of call_soon() in the coroutine runner rather than in
the Future itself. So asyncio Futures yielded in a coroutine that uses
Tornado's runner will be called one iteration later as they make two trips
through the scheduler, while if asyncio.Task gained the ability to
understand Tornado Futures they could turn into a busy loop that let
nothing else run. If we want interoperability, we're going to have to
mandate that code like SSLProtocol be tolerant of changes in timing (of
course, saying that is one thing and achieving it is another)


>
> In fact, I’m thinking now that Tornado/Twisted should keep their callbacks
> scheduling logic as is, and asyncio should keep its current implementation.
>

Yeah, the risks of making any changes in this area outweigh the benefits,
since we'll never get all the implementations to schedule things in exactly
the same way.

-Ben


>
>
> Anyways, I’ve tried to update asyncio Future to call callbacks directly:
>
> https://github.com/1st1/asyncio/commit/9da7ff868f1405893947118178d62bb7e1eb3a51
>
> The result is many broken unittests, until... a segfault in
> test_create_ssl_connection:
> https://gist.github.com/1st1/797dd12932bba7b353038fb7a80148ac (I’ll
> investigate this later).
>
> I’d be -1 to change callbacks scheduling in 3.5, IMO this has to wait
> until 3.6 even if we decide this should be done.
>
> [..]
> > +1 for replacing isinstance checks with an attribute that Twisted can
> add to Deferreds.
> > +1 for Ben’s suggestion that we should avoid ‘asyncio’ in the name here.
> __isfuture__?
>
> Yes, an asyncio-agnostic name would be better.  I like __isfuture__.
>
>
> Yury
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160604/72704490/attachment.html>