[Python-Dev] async/await in Python; v2

[Yury Selivanov]

Hi PJ,

On 2015-04-22 3:44 PM, PJ Eby wrote:
> On Tue, Apr 21, 2015 at 1:26 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>> It is an error to pass a regular context manager without ``__aenter__``
>> and ``__aexit__`` methods to ``async with``.  It is a ``SyntaxError``
>> to use ``async with`` outside of a coroutine.
> I find this a little weird.  Why not just have `with` and `for` inside
> a coroutine dynamically check the iterator or context manager, and
> either behave sync or async accordingly?  Why must there be a
> *syntactic* difference?

One of the things that we try to avoid is to have implicit
places where code execution might be suspended. For that
we use 'yield from' right now, and want to use 'await' with
PEP 492.

To have implicit context switches there is Stackless Python
and greenlets, however, it's harder to reason about the code
written in such a way.  Having explicit 'yield from/await'
is the selling point of asyncio and other frameworks that
use generator-based coroutines.

Hence, we want to stress that 'async with' and 'async for'
do suspend the execution in their protocols.

I don't want to loose control over what kind of iteration
or context manager I'm using.  I don't want to iterate
through a cursor that doesn't do prefetching, I want to
make sure that it does.  This problem is solved by the PEP.

>
> Not only would this simplify the syntax, it would also allow dropping
> the need for `async` to be a true keyword, since functions could be
> defined via "def async foo():" rather than "async def foo():"
>
> ...which, incidentally, highlights one of the things that's been
> bothering me about all this "async foo" stuff: "async def" looks like
> it *defines the function* asynchronously (as with "async with" and
> "async for"), rather than defining an asynchronous function.  ISTM it
> should be "def async bar():" or even "def bar() async:".

If we keep 'async with', then we'll have to keep 'async def'
to make it symmetric and easier to remember.  But, in theory,
I'd be OK with 'def async'.

'def name() async' is something that will be extremely hard
to notice in the code.

>
> Also, even that seems suspect to me: if `await` looks for an __await__
> method and simply returns the same object (synchronously) if the
> object doesn't have an await method, then your code sample that
> supposedly will fail if a function ceases to be a coroutine *will not
> actually fail*.

It doesn't just do that. In the reference implementation, a
single 'await o' compiles to:

(o) # await arg on top of the stack
GET_AWAITABLE
LOAD_CONST None
YIELD_FROM

Where GET_AWAITABLE does the following:

- If it's a coroutine-object -- return it
- If it's an object with __await__, return iter(object.__await__())
- Raise a TypeError of two above steps don't return

If you had a code like that:

    await coro()

where coro is

async def coro(): pass

you then can certainly refactor core to:

def coro(): return future # or some awaitable, please refer to PEP492

And it won't break anything.

So I'm not sure I understand your remark about
"*will not actually fail*".
>
> In my experience working with coroutine systems, making a system
> polymorphic (do something appropriate with what's given) and
> idempotent (don't do anything if what's wanted is already done) makes
> it more robust.  In particular, it eliminates the issue of mixing
> coroutines and non-coroutines.

Unfortunately, to completely eliminate the issue of reusing
existing "non-coroutine" code, or of writing "coroutine" code
that can be used with "non-coroutine" code, you have to use
gevent-kind of libraries.

>
> To sum up: I can see the use case for a new `await` distinguished from
> `yield`, but I don't see the need to create new syntax for everything;
> ISTM that adding the new asynchronous protocols and using them on
> demand is sufficient.  Marking a function asynchronous so it can use
> asynchronous iteration and context management seems reasonably useful,
> but I don't think it's terribly important for the type of function
> result.  Indeed, ISTM that the built-in `object` class could just
> implement `__await__` as a no-op returning self, and then *all*
> results are trivially asynchronous results and can be awaited
> idempotently, so that awaiting something that has already been waited
> for is a no-op.

I see all objects implementing __await__ returning "self" as
a very error prone approach.  It's totally OK to write code
like that:

async def coro():
    return fut
future = await coro()

In the above example, if coro ceases to be a coroutine,
'future' will be a result of 'fut', not 'fut' itself.

>   (Prior art: the Javascript Promise.resolve() method,
> which takes either a promise or a plain value and returns a promise,
> so that you can write code which is always-async in the presence of
> values that may already be known.)
>
> Finally, if the async for and with operations have to be distinguished
> by syntax at the point of use (vs. just always being used in
> coroutines), then ISTM that they should be `with async foo:` and `for
> async x in bar:`, since the asynchronousness is just an aspect of how
> the main keyword is executed.
>
> tl;dr: I like the overall ideas but hate the syntax and type
> segregation involved: declaring a function async at the top is OK to
> enable async with/for semantics and await expressions, but the rest
> seems unnecessary and bad for writing robust code.  (e.g. note that
> requiring different syntax means a function must either duplicate code
> or restrict its input types more, and type changes in remote parts of
> the program will propagate syntax changes throughout.)

Thanks,
Yury