[Python-ideas] Chaining asyncio futures?

[Cory Benfield]

> On 30 Jun 2016, at 08:23, Mark E. Haase <mehaase at gmail.com> wrote:
> 
> This does not make f2 dependent on f1; it immediately sets f2's result to repr(f1). It's easy to modify the example so that f1's callback explicitly sets f2's result. Abstracting a bit, we can write a chain_futures() function.
> 
>     def chain_futures(a, b):
>         a.add_done_callback(lambda _: b.set_result(a.result()))
> 
>     f3 = asyncio.Future()
>     f4 = asyncio.Future()
>     f4.add_done_callback(lambda f: print('done: {}'.format(f.result())))
>     chain_futures(f3, f4)
>     f3.set_result('hello world!')
> 
>     loop.run_until_complete(f4)
> 
> Output:
> 
>     done: hello world!
> 
> Wouldn't this be a nice feature in asyncio.Future? E.g. instead of `chain_futures(f3, f4)`, either `f3.set_result(f4)` or something more explicit like `f3.set_result_from(f4)`.

For a reference point: Twisted’s Deferreds have this already, written almost exactly in the form you’re considering it: https://twistedmatrix.com/documents/current/core/howto/defer.html#chaining-deferreds <https://twistedmatrix.com/documents/current/core/howto/defer.html#chaining-deferreds>. The biggest difference for Twisted is that it also hooks up the errbacks, which is a semantic that doesn’t make sense for asyncio Futures.

While discussing this, the natural way to write this kind of code in Twisted is actually to return Deferreds from a callback (the asyncio equivalent would be to return a Future from a done callback). In Twisted, if a callback returns a Deferred the next callback in the chain is not called until the Deferred resolves, at which point it is called with the result of that Deferred. Essentially, then, the program shown above returns only a single Deferred with a long series of callbacks that transform the data, some of which operate asynchronously.

As best as I can work out, the only reason that asyncio Future’s don’t behave this way is that asyncio strongly emphasises the use of coroutines to program in this manner. Put another way, the asyncio-native way to write that code is not to chain Futures, but instead to write a coroutine that manages your flow control for you:

    async def do_work():
        result = await check_cache(url)
        if not result:
            result = await do_web_request(url)
        print(‘done: {}’.format(f.result()))

    loop.run_until_complete(do_work())

This is not a reason in and of itself not to have chain_futures as a thing that exists. However, I think it may be the case that the asyncio core developers aren’t hugely interested in it. I’m now reading a bit into the mindset of Guido so I’m sure he could step in and correct me, but I seem to recall that asyncio’s Futures were deliberately designed to have less of the complexity of, say, Twisted’s Deferreds, in part because coroutines were intended to supersede some of the more complex functionality of Deferreds.

Cory

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160630/511d6593/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160630/511d6593/attachment-0001.sig>