[Python-ideas] New PEP 550: Execution Context

[Yury Selivanov]

On Sat, Aug 12, 2017 at 10:09 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 13 August 2017 at 03:53, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>> On Sat, Aug 12, 2017 at 1:09 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Now that you raise this point, I think it means that generators need
>>> to retain their current context inheritance behaviour, simply for
>>> backwards compatibility purposes. This means that the case we need to
>>> enable is the one where the generator *doesn't* dynamically adjust its
>>> execution context to match that of the calling function.
>>
>> Nobody *intentionally* iterates a generator manually in different
>> decimal contexts (or any other contexts). This is an extremely error
>> prone thing to do, because one refactoring of generator -- rearranging
>> yields -- would wreck your custom iteration/context logic. I don't
>> think that any real code relies on this, and I don't think that we are
>> breaking backwards compatibility here in any way. How many users need
>> about this?
>
> I think this is a reasonable stance for the PEP to take, but the
> hidden execution state around the "isolated or not" behaviour still
> bothers me.
>
> In some ways it reminds me of the way function parameters work: the
> bound parameters are effectively a *shallow* copy of the passed
> arguments, so callers can decide whether or not they want the callee
> to be able to modify them based on the arguments' mutability (or lack
> thereof).

Mutable default values for function arguments is one of the most
confusing things to its users.  I've seen numerous threads on
StackOverflow/Reddit with people complaining about it.

> That similarity makes me wonder whether the "isolated or not"
> behaviour could be moved from the object being executed and directly
> into the key/value pairs themselves based on whether or not the values
> were mutable, as that's the way function calls work: if the argument
> is immutable, the callee *can't* change it, while if it's mutable, the
> callee can mutate it, but it still can't rebind it to refer to a
> different object.

I'm afraid that if we design EC context to behave differently for
mutable/immutable values, it will be an even harder thing to
understand to end users.

> 1. If a parent context wants child contexts to be able to make
> changes, then it should put a *mutable* object in the context (e.g. a
> list or class instance)
> 2. If a parent context *does not* want child contexts to be able to
> make changes, then it should put an *immutable* object in the context
> (e.g. a tuple or number)
> 3. If a child context *wants to share a context key with its parent,
> then it should *mutate* it in place
> 4. If a child context *does not* want to share a context key with its
> parent, then it should *rebind* it to a different object

It's possible to put mutable values even with the current PEP 550 API.
The issue that Nathaniel has with it, is that he actually wants the
API to behave exactly like it does to implement his timeouts logic,
but: there's a corner case, where isolating generator state at the
time when it is created doesn't work in his favor.

FWIW I believe that I now have a complete solution for the
generator.send() problem that will make it possible for Nathaniel to
implement his Trio APIs.

The functional PoC is here: https://github.com/1st1/cpython/tree/pep550_gen

The key change is to make generators and asynchronous generators to:

1. Have their own empty execution context when created. It will be
used for whatever local modifications they do to it, ensuring that
their state never escapes to the outside world
(gi_isolated_execution_context flag is still here for contextmanager).

2. ExecutionContext has a new internal pointer called ec_back. In the
Generator.send/throw method, ec_back is dynamically set to the current
execution context.

3. This makes it possible for generators to see any outside changes in
the execution context *and* have their own, where they can make
*local* changes.

So (pseudo-code):

    def gen():
        print('1', context)
        yield
        print('2', context)
        with context(spam=ham):
             yield
             print('3', context)
             yield
        print('4', context)
        yield

    g = gen()
    context(foo=1, spam='bar')
    next(g)
    context(foo=2)
    next(g)
    context(foo=3)
    next(g)
    context(foo=4)
    next(g)

will print:

    1 {foo=1, spam=bar}
    2 {foo=2, spam=bar}
    3 {foo=3, spam=ham}
    4 {foo=4, spam=bar}

There are some downsides to the approach, mainly from the performance
standpoint, but in a common case they will be negligible, if
detectable at all.

Yury