asyncio: What is the difference between tasks, futures, and coroutines?

[Paul Moore]

On Tuesday, 5 May 2015 17:11:39 UTC+1, Zachary Ware  wrote:
>On Tue, May 5, 2015 at 10:22 AM, Paul  Moore wrote:
>> I'm working my way through the asyncio documentation. I have got to the
>> "Tasks and coroutines" section, but I'm frankly confused as to the
>> difference between the various things described in that section:
>> coroutines, tasks, and futures.
> 
> I've been using (and, in part at least, understanding :)) asyncio for
> work for the past few months, so I can at least give you my
> impressions based on use.

Thanks for taking the time.
 
>> I think can understand a coroutine. Correct me if I'm wrong, but it's
>> roughly "something that you can run which can suspend itself".
> 
> That's basically how I understand it.

Cool, I got that right at least :-)

>> But I don't understand what a Future is.
[...]
> 
> My understanding is that Futures are somewhat like 'non-executable
> coroutines', if that makes any sense whatsoever.  Futures are used as
> something you can pass around when you need to start execution of the
> "job" in one method and finish it in another.  For instance, talking
> to a remote network entity, and you want to just do "yield from
> server.get(something)".  Your 'get' method can make the request,
> create a Future, stick it somewhere that the method monitoring
> incoming traffic can find it (along with some identifying metadata),
> and then do 'return (yield from future)' to wait for the Future to be
> fulfilled (by the listener) and return its result.  The listener then
> matches up incoming requests with Futures, and calls
> Future.set_result() or Future.set_exception() to fulfill the Future.
> Once one of those methods has been called on the Future (and control
> has been passed back to the scheduler), your server.get method
> unblocks from its '(yield from future)' call, and either raises the
> exception or returns the result that was set on the Future.

Hmm. My head hurts. I sort of think I see what you might mean, but I have no idea how that translates to "almost the same as a concurrent.futures.Future, which in my experience is nothing like that.

A c.f.Future is (in my naive view) a "promise" of a result, which you can later wait to be fulfilled. Creating one directly (as opposed to via submit) doesn't make sense because then there's nothing that's promised to provide a result.

What you described sounds more like a "container that can signal when it's been filled", which I can sort of see as related to the Future API, but not really as related to a c.f.Future.

>> Reading between the lines, it seems that the event loop schedules Tasks
>> (which makes sense) and that Tasks somehow wrap up coroutines - but I
>> don't see *why* you need to wrap a task in a coroutine rather than just
>> scheduling coroutines. And I don't see where Futures fit in - why not
>> just wrap a coroutine in a Future, if it needs to be wrapped up at all?
> 
> You kind of mixed things up in this paragraph (you said "Tasks somehow
> wrap up coroutines", then "wrap a task in a coroutine"; the first is
> more correct, I believe).

Whoops, sorry, my mistake - the first was what I meant, the second was a typo.

> As I understand it, Tasks are
> specializations of Futures that take care of the
> set_result/set_exception based on the execution of the coroutine.

That sounds far more like a concurrent.futures.Future. Except that (AIUI) tasks get scheduled, and you can list all tasks for an event loop, and things like that. So they seem like a little bit more than a c.f.Future, which would sort of match with the idea that an asyncio.Future was equivalent to a c.f.Future, except that it's not (as above).

There's also things like asyncio.async - which makes no sense to me at all (I can see how the examples *use* it, in much the same way as I can see how the examples work, in general, but I can't understand its role, so I'm not sure I'd be able to design code that used it from scratch :-()

>> Can anyone clarify for me?
> 
> I hope I've done some good on that front, but I'm still a bit hazy
> myself.  I found it worked best to just take examples or otherwise
> working code, and poke and prod at it until it breaks and you can
> figure out how you broke it.  Just try to avoid mixing asyncio and
> threads for as long as you can :).

Thanks for the insights. I think I've learned some things, but to be honest I'm still confused. Maybe it's because of the angle I'm coming at it from. I don't typically write network code directly[1], so the examples in the asyncio docs are mostly irrelevant at best for me. I do find that I use threads a reasonable amount, and I'd like to know if asyncio would be a worthwhile alternative. So I need to understand the concepts and try to apply them to my situations. Hence the comment above about designing asyncio code from scratch.

Paul

[1] When I do write network code, I typically want to use a high level library like requests. So in an asyncio context, I'm looking at how to use an existing, synchronous library, in async code. In twisted, I'd use defer_to_thread, and I think run_in_executor is the equivalent asyncio concept. The docs say that is a coroutine, so I probably need to wrap that in a task to schedule it, but that's where a proper understanding of the concepts becomes crucial...