[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 14:15, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> On 22 November 2017 at 15:09, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> On 22 November 2017 at 13:53, Ivan Levkivskyi <levkivskyi at gmail.com>
>> wrote:
>> > On 22 November 2017 at 14:38, Antoine Pitrou <solipsis at pitrou.net>
>> > wrote:
>> >>
>> >> On Wed, 22 Nov 2017 15:03:09 +0200
>> >> Serhiy Storchaka <storchaka at gmail.com> wrote:
>> >> >  From
>> >> >
>> >> >
>> >> > https://stackoverflow.com/questions/45190729/differences-between-generator-comprehension-expressions.
>> >> >
>> >> >      g = [(yield i) for i in range(3)]
>> >> >
>> >> > Syntactically this looks like a list comprehension, and g should be a
>> >> > list, right? But actually it is a generator. This code is equivalent
>> >> > to
>> >> > the following code:
>> >> >
>> >> >      def _make_list(it):
>> >> >          result = []
>> >> >          for i in it:
>> >> >              result.append(yield i)
>> >> >          return result
>> >> >      g = _make_list(iter(range(3)))
>> >> >
>> >> > Due to "yield" in the expression _make_list() is not a function
>> >> > returning a list, but a generator function returning a generator.
>> >> >
>> >> > This change in semantic looks unintentional to me. It looks like
>> >> > leaking
>> >> > an implementation detail.
>> >>
>> >> Perhaps we can deprecate the use of "yield" in comprehensions and make
>> >> it a syntax error in a couple versions?
>> >>
>> >> I don't see a reason for writing such code rather than the more
>> >> explicit variants.  It looks really obscure, regardless of the actual
>> >> semantics.
>> >
>> >
>> > People actually try this (probably simply because they like
>> > comprehensions)
>> > see two mentioned Stackoverflow questions, plus there are two b.p.o.
>> > issues.
>> > So this will be a breaking change. Second, recent PEP 530 allowed
>> > writing a
>> > similar comprehensions with `await`:
>> >
>> >     async def process(funcs):
>> >         result = [await fun() for fun in funcs]  # OK
>> >         ...
>> >
>> > Moreover, it has the semantics very similar to the proposed by Serhiy
>> > for
>> > `yield` (i.e. equivalent to for-loop without name leaking into outer
>> > scope).
>> > Taking into account that the actual fix is not so hard, I don't think it
>> > makes sense to have all the hassles of deprecation period.
>>
>> I agree with Antoine. This (yield in a comprehension) seems far too
>> tricky, and I'd rather it just be rejected. If I saw it in a code
>> review, I'd certainly insist it be rewritten as an explicit loop.
>>
>
> There are many things that I would reject in code review, but they are still
> allowed in Python,
> this is one of the reasons why code reviews exist. Also I am not sure how
> `yield` in a comprehension
> is more tricky than `await` in a comprehension. Anyway, this looks more like
> a taste question.

I generally don't understand "await" in any context, so I deferred
judgement on that :-) Based on your comment that they are equally
tricky, I'd suggest we prohibit them both ;-)

Less facetiously, comprehensions are defined in the language reference
in terms of a source translation to nested loops. That description
isn't 100% precise, but nevertheless, if yield/async in a
comprehension doesn't behave like that, I'd consider it a bug. So
current behaviour (for both yield and await) is a bug, and your
proposed semantics for yield is correct. Await in a comprehension
should work similarly - the docs for await expressions just say "Can
only be used inside a coroutine function", so based on that they are
legal within a comprehension inside a coroutine function (if the
semantics in that case isn't obvious, it should be clarified - the
await expression docs are terse to the point that I can't tell
myself).

So I'm +1 on your suggestion.

I remain concerned that yield/await expressions are adding a lot of
complexity to the language, that isn't explained in the manuals in a
way that's accessible to non-specialists. Maybe "ban (or reject at
code review) all the complex stuff" isn't a reasonable approach, but
nor is expecting everyone encountering async code to have read and
understood all the async PEPs and docs. We need to consider
maintenance programmers as well (who are often looking at code with
only a relatively high-level understanding).

Paul