[Python-Dev] async/await in Python; v2

[Paul Sokolovsky]

Hello,

On Wed, 22 Apr 2015 09:53:39 -0700
Rajiv Kumar <rajiv.kumar at gmail.com> wrote:

> I'd like to suggest another way around some of the issues here, with
> apologies if this has already been discussed sometime in the past.
> 
> From the viewpoint of a Python programmer, there are two distinct
> reasons for wanting to suspend execution in a block of code:
> 
> 1. To yield a value from an iterator, as Python generators do today.
> 
> 2. To cede control to the event loop while waiting for an
> asynchronous task to make progress in a coroutine.
> 
> As of today both of these reasons to suspend are supported by the same
> underlying mechanism, i.e. a "yield" at the end of the chain of "yield
> from"s. PEPs 492 and 3152 introduce "await" and "cocall", but at the
> bottom of it all there's effectively still a yield as I understand it.
> 
> I think that the fact that these two concepts use the same mechanism
> is what leads to the issues with coroutine-generators that Greg and
> Yury have raised.
> 
> With that in mind, would it be possible to introduce a second form of
> suspension to Python to specifically handle the case of ceding to the
> event loop? 

Barring adding an adhoc statement "yield_to_a_main_loop", there's a
generic programming device to do it: symmetric coroutines. But it's
unlikely to help with your sentiment that the same device is used for
different purposes. At least with asymmetric coroutines as currently in
Python, you have next() to "call" a coroutine and yield to "return"
from. With symmetric coroutines, you don't have a place to return -
you can only "call" another coroutine, and then have freedom to call
any (including a main loop), but need to bother to always know whom you
want to call.

But I guess it's already sounds confusing enough for folks who didn't
hear about symmetric coroutines, whereas call/return paradigm is much
more familiar and understandable. That's certainly why Python
implements asymmetric model. And having both asymmetric and symmetric
would quite confusing, especially that symmetric are more powerful and
asymmetric can be easily implemented in terms of symmetric using
continuation-passing style. On the last occurrence of "easily" mere
users of course start to run away, shouting that if they wanted to
use Scheme, they'd have taken classes on it and used long ago.   


So, the real problem with dichotomy you describe above is not
technical, but rather educational/documentational. And the current
approach asyncio takes is "you should not care if a coroutine yields
to main loop, or how it is done". Actually, the current approach is to
forbid and deny you knowledge how it is done, quoting Victor Stinner
from another mail: "(A) asyncio coroutine in Python 3.4: use yield
from, yield denied". So, just pretend that there's no yield, only
yield from, problem solved. But people know there's yield - they
knew it for long time before "yield from". And there're valid usages
for yield in a coroutine, like implementing your, application-level
generator/generation. Currently, any generation ability is usurped by
asyncio's main loop.

Much better approach IMHO is given in David Beazley's presentations on
generators and coroutines, http://www.dabeaz.com/generators/ . He says
that coroutines provided by framework are essentially "system calls".
And that's why you don't want to know how they work, and shouldn't
care - because users usually don't care how OS kernel implements
system calls while sitting in the web browser. But if you want, you can,
and you will discover that they're implemented by yield'ing objects
of a special class. That's why you're *suggested* not to use yield's in
coroutines - because if you want to catch yours, application-level,
yields, you may also get any time a system yield object. You would need
to expect such possibility, filter such yields and pass them up
(re-yield). But there's no forbidden magic in all that, and
understanding that helps a lot IMHO.



-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com