[Async-sig] asyncio <-> twisted interoperability

[Łukasz Langa]

Being able to re-use Twisted’s vast array of existing protocols and related libraries within asyncio would be a dream come true. Likewise, being able to write Twisted code using async/await coroutines without wrappers in-between would increase Twisted adoption since that syntax is more welcoming than callback-based Deferreds.

I have a few inline comments here.

> On Jun 3, 2016, at 1:11 PM, Yury Selivanov <yselivanov at gmail.com> wrote:
> 
> 2.1. There should be a way for Deferred objects to get the currently running event loop reliably. There is a pending PR to add the new ‘asyncio.get_running_loop’ function.  If we land it in 3.5.2, Deferred object will be able to get the currently running event loop and schedule callbacks in a fair fashion, similarly to Futures.

In general, writing polyglot code sucks. I always recommend sticking to a single version for applications. As library maintainers, we’re the poor people that have to provide compatibility across versions, typically with Tox.

I don’t know to what extent we can expect coroutines to be FULLY COMPATIBLE across async frameworks. They should be REUSABLE but that’s not the same thing. Specifically, having Deferreds calling their callbacks directly is perfectly fine by me. The goal here is to reuse what the Twisted ecosystem has already implemented so expecting all deferreds to now schedule callbacks on the event loop is not realistic. In the same vein, we can expect some coroutines to reuse asyncio internals (commingling with the event loop’s implementation details for instance) to the extent where compatibility with Twisted might not be possible.

Actually, +1 for Ben’s suggestion to revisit if asyncio.Futures really need call_soon scheduling of callbacks. Specifically, a torrent of call_soon doesn’t really achieve “fair scheduling” as suggested by Yury anyway:

https://gist.github.com/ambv/3324e3e569b9f7883dfcb9c8cb1d5445

There’s still a comment on the asyncio codebase suggesting we might be able to merge concurrent.futures.Future with asyncio.Future. This would be a worthwhile goal.

> 2.2. asyncio uses ‘isinstance(o, Future)’ calls in many key places, such as ‘asyncio.ensure_future’ and in the Task class.  The latter makes it impossible for anything but Future and coroutines to be awaitable in asyncio code.  We have two options to fix this: make Future an instance of ABCMeta, or introduce a new protocol for Future-like objects.  The former option will not be considered because it makes isinstance calls much slower.
> 
> As for the second option — a protocol for Future-like objects — I propose to add a new dunder attribute ‘__asyncio_future__ = True’ to asyncio.Future class, and replace all ‘isinstance(o, Future)’ calls with ‘hasattr(o, “__asyncio_future__”)’ checks.
> 

+1 for replacing isinstance checks with an attribute that Twisted can add to Deferreds.
+1 for Ben’s suggestion that we should avoid ‘asyncio’ in the name here. __isfuture__?


> - I don’t think we need ‘twisted.deferredCoroutine’ decorator — that will make async/await a lot slower. Instead, Twisted should follow asyncio and create an alternative, tailored for Twisted, implementation of ‘asyncio.Task’, along with ‘ensure_deferred’ and/or ‘create_task’ APIs.  This way users can create Deferreds for coroutines only when they need it.

+1 here. Having to decorate async/await coroutines to be able to use them within Twisted is an adoption barrier. The same would be true about having to decorate Deferreds before awaiting on them in asyncio.

Amber, Glyph, Ben, is asyncio.Future’s event loop callback scheduling the biggest barrier in interoperability at the moment?

- Ł
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/async-sig/attachments/20160603/b3d91e33/attachment.sig>