[Python-ideas] PEP 530: Asynchronous Comprehensions

[Nick Coghlan]

On 6 September 2016 at 11:22, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> On 2016-09-05 3:57 PM, Ivan Levkivskyi wrote:
>> There is an old "bug" (some people call this a feature)
>> http://bugs.python.org/issue10544
>> If one uses yield in a comprehension, then it leads to unexpected results:
>
> I'm not sure what will happen here. Will have an update once a reference
> implementation is ready.  Seems that the bug you're referring to is
> something that should be just fixed.

I was going to say that the problem with that last sentence is the
"just" part, as for a long time, even though the more desirable
behaviour was relatively clear ("make it work the way it did in Python
2"), we didn't have a readily available means of doing that. However,
the last time I thought seriously about this problem was before we
added "yield from", and that may actually be enough to change the
situation.

Specifically, the problem is that comprehensions and generator
expressions in 3.x create a full closure, so code like:

    def example():
            L = [(yield) for i in range(2)]
            print(L)

is implicitly doing:

    def example():
        def _bad_gen(_arg):
            result = []
            for i in _arg:
                result.append((yield))
            return result
        L = _bad_gen(range(2))
        print(L)

Prior to yield from, the code generator had no easy way to fix that,
since it couldn't delegate yield operations to the underlying
generator. However, given PEP 380, the code generator could
potentially detect these situations, and implicitly use "yield from"
to hoist the generator behaviour up to the level of the containing
function, exactly as happened in Python 2:

    def example():
        def _nested_gen(_arg):
            result = []
            for i in _arg:
                result.append((yield))
            return result
        L = yield from _nested_gen(range(2))
        print(L)

>>> gen = example()
>>> next(gen)
>>> gen.send(1)
>>> gen.send(2)
[1, 2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

I don't think there's anything we can do about generator expressions
short of PEP 530 itself though - they already misbehave in Python 2,
and misbehave in exactly the same way in Python 3, since there's no
way for the interpreter to tell the difference at runtime between the
implicit yields from the generator expression itself, and any explicit
yields used in generator subexpressions.

By contrast, PEP 530 can work, since it doesn't *want* to tell the
difference between the implicit awaits and explicit awaits, and the
await and yield channels are already distinguished at the language
level.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia