[Python-Dev] Tricky way of of creating a generator via a comprehension expression

[Paul Moore]

On 22 November 2017 at 16:47, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> On 22 November 2017 at 17:43, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> On 22 November 2017 at 16:30, Ivan Levkivskyi <levkivskyi at gmail.com>
>> wrote:
>> > On 22 November 2017 at 17:24, Antoine Pitrou <solipsis at pitrou.net>
>> > wrote:
>> >> Given a comprehension (e.g. list comprehension) is expected to work
>> >> nominally as `constructor(generator expression)`
>> >
>> > As Yury just explained, these two are not equivalent if there is an
>> > `await`
>> > in the comprehension/generator expression.
>>
>> As Antoine said, people *expect* them to work the same.
>
>
> The difference is that a generator expression can be used independently, one
> can assign it to a variable etc. not necessary to wrap it into a list()
> Anyway, can you propose an equivalent "defining" code for both? Otherwise it
> is not clear what you are defending.

I can't propose a precise description of the semantics in the form "(x
for x in expr) is precisely equivalent to <...>" (for generator
expressions or list/dict/set comprehensions). I'm not even sure that
there *is* a definition in those terms - I don't see it as a given
that generator expressions are simply syntax shorthand. It's not
necessary that such an expansion exists - there's no such expansion
for most expressions in Python. The equivalences I would suggest,
you've stated previously in this thread are inaccurate, so it's not
clear to me what options you've left open for me.

What I can say is that the semantics of generator expressions and
comprehensions should match the understanding that has been documented
and taught for many years now. Or at least, if the semantics change,
then documentation, "what's new in Python x.y", and tutorials should
be updated to match. In my view, the following understandings have
been true since comprehensions and generators were introduced, and
should not be changed without careful management of the change. (I
accept that things *have* changed, so we have a retroactive exercise
that we need to do to mitigate any damage to people's understanding
from the changes, but I'm trying to start from what I believe is the
baseline understanding that people have had since pre-async, and don't
yet understand how to modify).

1. List comprehensions expand into nested for/if statements in the
"obvious" way - with an empty list created to start and append used to
add items to it.
   1a. Variables introduced in the comprehension don't "leak" (see below).
2. Generator expressions expand into generators with the same "nested
loop" behaviour, and a yield of the generated value.
3. List comprehensions are the same as list(the equivalent generator
expression).

With regard to 1a, note that Python 2 *did* leak names. The change to
stop those leaks is arguably a minor, low-level detail (much like the
sort of things coming up in regard to yield expressions and async).
But it was considered significant enough to go through a long and
careful debate, and a non-trivial amount of publicity ensuring users
were aware of, and understood, the change. I see that as a good
example of how the semantics *can* change, but we should ensure we
manage users' understanding.

I'm not sure we're going to reach agreement on this. Let it stand that
I believe the above 3 points are key to how people understand
comprehensions/generator expressions, and I find the subtle
discrepancies that have been introduced by async to be difficult to
understand, and counter to my intuition. I don't believe I'm alone in
this.

I can't tell you how to explain this clearly to me - by definition, I
don't know a good explanation. I take it as self-evident that having a
big chunk of Python semantics that leaves people thinking "I'll avoid
this because it's confusing" is bad - maybe you (or others) see it as
simply "advanced concepts" that I (and people like me) can ignore, but
while I usually do just that, threads like this keep cropping up for
which that approach doesn't work...

Paul