[Python-Dev] PEP 550 v4: coroutine policy

[Yury Selivanov]

On Tue, Aug 29, 2017 at 3:32 PM, Antoine Pitrou <antoine at python.org> wrote:
>
>
> Le 29/08/2017 à 21:18, Yury Selivanov a écrit :
>> On Tue, Aug 29, 2017 at 2:40 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>> On Mon, 28 Aug 2017 17:24:29 -0400
>>> Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>>>> Long story short, I think we need to rollback our last decision to
>>>> prohibit context propagation up the call stack in coroutines.  In PEP
>>>> 550 v3 and earlier, the following snippet would work just fine:
>>>>
>>>>    var = new_context_var()
>>>>
>>>>    async def bar():
>>>>        var.set(42)
>>>>
>>>>    async def foo():
>>>>        await bar()
>>>>        assert var.get() == 42   # with previous PEP 550 semantics
>>>>
>>>>    run_until_complete(foo())
>>>>
>>>> But it would break if a user wrapped "await bar()" with "wait_for()":
>>>>
>>>>    var = new_context_var()
>>>>
>>>>    async def bar():
>>>>        var.set(42)
>>>>
>>>>    async def foo():
>>>>        await wait_for(bar(), 1)
>>>>        assert var.get() == 42  # AssertionError !!!
>>>>
>>>>    run_until_complete(foo())
>>>>
>>> [...]
>>
>>> Why wouldn't the bar() coroutine inherit
>>> the LC at the point it's instantiated (i.e. where the synchronous bar()
>>> call is done)?
>>
>> We want tasks to have their own isolated contexts.  When a task
>> is started, it runs its code in parallel with its "parent" task.
>
> I'm sorry, but I don't understand what it all means.
>
> To pose the question differently: why is example #1 supposed to be
> different, philosophically, than example #2?  Both spawn a coroutine,
> both wait for its execution to end.  There is no reason that adding a
> wait_for() intermediary (presumably because the user wants to add a
> timeout) would significantly change the execution semantics of bar().

I see your point. The currently published version of the PEP (v4)
fixes this by saying: each coroutine has its own LC. Therefore,
"var.set(42)" cannot be visible to the code that calls "bar()".  And
therefore, "await wait_for(bar())" and "await bar()" work the same way
with regards to execution context semantics.

*Unfortunately*, while this fixes above examples to work the same way,
setting context vars in "__aenter__" stops working:

     class MyAsyncCM:

             def __aenter__(self):
                    var.set(42)

     async with MyAsyncCM():
           assert var.get() == 42

Because __aenter__ has its own LC, the code wrapped in "async with"
will not see the effect of "var.set(42)"!

This absolutely needs to be fixed, and the only way (that I know) it
can be fixed is to revert the "every coroutine has its own LC"
statement (going back to the semantics coroutines had in PEP 550 v2
and v3).

>
>> wait_for() in the above example creates an asyncio.Task implicitly,
>> and that's why we don't see 'var' changed to '42' in foo().
>
> I don't understand why a non-obvious behaviour detail (the fact that
> wait_for() creates an asyncio.Task implicitly) should translate into a
> fundamental difference in observable behaviour.  I find it
> counter-intuitive and error-prone.

"await bar()" and "await wait_for(bar())" are actually quite
different.  Let me illustrate with an example:

    b1 = bar()
    # bar() is not running yet
    await b1

    b2 = wait_for(bar())
    # bar() was wrapped into a Task and is being running right now
    await b2

Usually this difference is subtle, but in asyncio it's perfectly fine
to never await on b2, just let it run until it completes.  If you
don't "await b1" -- b1 simply will never run.

All in all, we can't say that "await bar()" and "await
wait_for(bar())" are equivalent.  The former runs bar() synchronously
within the coroutine that awaits it.  The latter runs bar() in a
completely separate and detached task in parallel to the coroutine
that spawned it.

>
>> This is a slightly complicated case, but it's addressable with a good
>> documentation and recommended best practices.
>
> It would be better addressed with consistent behaviour that doesn't rely
> on specialist knowledge, though :-/

I agree. But I don't see any other solution that would solve the
problem *and* satisfy the following requirements:

1. Context variables set in "CM.__aenter__" and "CM.__aexit__" should
be visible to code that is wrapped in "async with CM()".

2. Tasks must have isolated contexts -- changes that coroutines do to
the EC in one Task, should not be visible to other Tasks.

Yury