[Python-Dev] Using async/await in place of yield expression

[Nathaniel Smith]

On Sun, Nov 26, 2017 at 9:33 PM, Caleb Hattingh
<caleb.hattingh at gmail.com> wrote:
> The PEP only says that __await__ must return an iterator, but it turns out
> that it's also required that that iterator
> should not return any intermediate values.

I think you're confused :-). When the iterator yields an intermediate
value, it does two things:

(1) it suspends the current call stack and returns control to the
coroutine runner (i.e. the event loop)
(2) it sends some arbitrary value back to the coroutine runner

The whole point of `await` is that it can do (1) -- this is what lets
you switch between executing different tasks, so they can pretend to
execute in parallel. However, you do need to make sure that your
__await__ and your coroutine runner are on the same page with respect
to (2) -- if you send a value that the coroutine runner isn't
expecting, it'll get confused. Generally async libraries control both
the coroutine runner and the __await__ method, so they get to invent
whatever arbitrary convention they want.

In asyncio, the convention is that the values you send back must be
Future objects, and the coroutine runner interprets this as a request
to wait for the Future to be resolved, and then resume the current
call stack. In curio, the convention is that you send back a special
tuple describing some operation you want the event loop to perform
[1], and then it resumes your call stack once that operation has
finished. And Trio barely uses this channel at all. (It does transfer
a bit of information that way for convenience/speed, but the main work
of setting up the task to be resumed at the appropriate time happens
through other mechanisms.)

What you observed is that the asyncio coroutine runner gets cranky if
you send it an integer when it was expecting a Future.

Since most libraries assume that they control both __await__ and the
coroutine runner, they don't tend to give great error messages here
(though trio does [2] ;-)). I think this is also why the asyncio docs
don't talk about this. I guess in asyncio's case it is technically a
semi-public API because you need to know how it works if you're the
author of a library like tornado or twisted that wants to integrate
with asyncio. But most people aren't the authors of tornado or
twisted, and the ones who are already know how this works, so the lack
of docs isn't a huge deal in practice...

-n

[1] https://github.com/dabeaz/curio/blob/bd0e2cb7741278d1d9288780127dc0807b1aa5b1/curio/traps.py#L48-L156
[2] https://github.com/python-trio/trio/blob/2b8e297e544088b98ff758d37c7ad84f74c3f2f5/trio/_core/_run.py#L1521-L1530

-- 
Nathaniel J. Smith -- https://vorpus.org