[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Nick Coghlan]

On 25 November 2017 at 13:30, Guido van Rossum <guido at python.org> wrote:
>
> On Fri, Nov 24, 2017 at 4:22 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> The more I hear about this topic, the more I think that `await`, `yield` and `yield from` should all be banned from occurring in all comprehensions and generator expressions. That's not much different from disallowing `return` or `break`.
>
>
> From the responses it seems that I tried to simplify things too far. Let's say that `await` in comprehensions is fine, as long as that comprehension is contained in an `async def`. While we *could* save `yield [from]` in comprehensions, I still see it as mostly a source of confusion, and the fact that the presence of `yield [from]` *implicitly* makes the surrounding `def` a generator makes things worse. It just requires too many mental contortions to figure out what it does.

While I'm OK with ruling the interaction of all of these
subexpressions with generator expression too confusingly weird to be
supportable, in https://bugs.python.org/issue10544?#msg306940 I came
up with an example for the existing comprehension behaviour that I'm
hesitant to break without a clear replacement: code that wraps the
*current* comprehension behaviour in an explicit "yield from"
expression.

That is, code like:

    def example():
        comp1 = yield from [(yield x) for x in ('1st', '2nd')]
        comp2 = yield from [(yield x) for x in ('3rd', '4th')]
        return comp1, comp2

Creates a generator that returns a 2-tuple containing two 2-item
lists, but the exact contents of those lists depend on the values you
send into the generator while it is running.

The basic principle behind the construct is that a comprehension
containing a yield expression essentially defines an inline
subgenerator, so you then have to yield from that subgenerator in
order to produce the desired list.

If the implicit "yield from" idea seems too magical, then the other
direction we could go is to make the immediate "yield from" mandatory,
such that leaving it out produces a SyntaxWarning in 3.7, and then a
SyntaxError in 3.8+. (Something like "'yield from' clause missing
before subgenerator comprehension")

While we don't generally like bolting two pieces of syntax together
like that, I think it's defensible in this case based on the fact that
allowing "[expr for var in iterable]" to ever return anything other
than a list (and similarly for the other comprehension types) is
genuinely confusing.

The nicest aspect of the "an explicit and immediate 'yield from' is
required when using subgenerator comprehensions" approach is that the
above syntax already works today, and has actually worked since 3.3 -
we'd just never put the pieces together properly to establish this as
a potential pattern for comprehension use in coroutines.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia