[Async-sig] asyncio <-> twisted interoperability

[Ben Darnell]

On Fri, Jun 3, 2016 at 4:11 PM, Yury Selivanov <yselivanov at gmail.com> wrote:

> During the asyncio open space a few days ago we had a number of important
> questions raised, including Twisted/asyncio interop.  I’d like to start the
> discussion of a few related things:
>

For those who haven't seen it, it's already possible to mix asyncio,
twisted, and tornado freely in tornado coroutines (using the `yield`
syntax. `await` can be used with tornado and asyncio, but not twisted last
time I looked):
https://groups.google.com/forum/#!topic/python-tulip/W9_Cj4zN1Jc


>
> 1. key difference between a Deferred and a Future;
> 2. make Deferred objects compatible with asyncio code;
> 3. overall strategy to make Twisted code compatible with asyncio code.
>
> In detail:
>
> 1. Unlike Futures, Deferred objects know nothing about the event loop.
> Deferred calls its callbacks directly, while Future uses ‘loop.call_soon’
> to ensure fair scheduling.
>

asyncio Futures know about their event loop, but concurrent futures and
Tornado futures do not. asyncio is the odd one out here, and I think it
would be better to make asyncio futures call their callbacks directly than
to introduce knowledge of the event loop to other deferred/future-like
objects.


>
> 2. I see two issues with Deferred/Future compatibility:
>
> 2.1. There should be a way for Deferred objects to get the currently
> running event loop reliably. There is a pending PR to add the new
> ‘asyncio.get_running_loop’ function.  If we land it in 3.5.2, Deferred
> object will be able to get the currently running event loop and schedule
> callbacks in a fair fashion, similarly to Futures.
>
> 2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such
> as ‘asyncio.ensure_future’ and in the Task class.  The latter makes it
> impossible for anything but Future and coroutines to be awaitable in
> asyncio code.  We have two options to fix this: make Future an instance of
> ABCMeta, or introduce a new protocol for Future-like objects.  The former
> option will not be considered because it makes isinstance calls much slower.
>

+1. This was my goal in the thread linked above.


>
> As for the second option — a protocol for Future-like objects — I propose
> to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future
> class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o,
> “__asyncio_future__”)’ checks.
>

Why make it asyncio specific? What you need is A) does it have an
add_done_callback() method with the right interface, and B) are callbacks
guaranteed to be run on the same thread (as with asyncio Futures) or is the
thread unspecified (as in concurrent Futures). The latter isn't really a
type-level check, so maybe Future instances should be associated with a
thread (rather than an event loop).


>
> (2.1) and (2.2) should make it possible for Twisted to make Deferred
> objects fully compatible with asyncio code.  Glyph and Amber, please
> confirm this.
>
> 3. I’ve been on Amber’s talk about Twisted and asyncio, and wanted to
> share my ideas on how Twisted should approach compatibility with asyncio
> and async/await syntax:
>
> - I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will
> make async/await a lot slower. Instead, Twisted should follow asyncio and
> create an alternative, tailored for Twisted, implementation of
> ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs.
> This way users can create Deferreds for coroutines only when they need it.


> - ‘ensure_deferred’ could also accept instances of ‘asyncio.Future’, and
> wrap them in a Deferred object.  This should make it possible to re-use
> asyncio-specific code in Twisted code.
>

This is what Tornado's `convert_yielded` function does to allow twisted and
asyncio libraries to be used from tornado applications. (I can't remember
why we have to wrap asyncio futures in tornado futures; the interfaces are
compatible except for tornado's extensions to better capture stack traces
on python 2).

-Ben


>
> Thoughts?
>
> Thanks,
> Yury
> _______________________________________________
> Async-sig mailing list
> Async-sig at python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/3e8a991f/attachment.html>