[Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part)

[Guido van Rossum]

[This is my one response today]

On Mon, Jun 25, 2018 at 12:40 PM Terry Reedy <tjreedy at udel.edu> wrote:

> On 6/24/2018 7:25 PM, Guido van Rossum wrote:
> > I'd wager that the people who might be most horrified about it
>
> the (b) scoping rule change
>
> > would be people who feel strongly that the change to the
> > comprehension scope rules in Python 3 is a big improvement,
>
> I might not be one of those 'most horrified' by (b), but I increasingly
> don't like it, and I was at best -0 on the comprehension scope change.
> To me, iteration variable assignment in the current scope is a
> non-problem.  So to me the change was mostly useless churn.  Little
> benefit, little harm.  And not worth fighting when others saw a benefit.
>

Fair enough, and by itself this might not have been enough reason to make
the change. But see below.


> However, having made the change to nested scopes, I think we should
> stick with them.  Or repeal them.  (I believe there is another way to
> isolate iteration names -- see  below).  To me, (b) amounts to half
> repealing the nested scope change, making comprehensions half-fowl,
> half-fish chimeras.
>

That depends on how you see it -- to me (b) just means that there's an
implicit nonlocal[1] to make the assignment have the (desirable)
side-effect.

The key thing to consider here is whether that side-effect is in fact
desirable. For me, the side-effect of the comprehension's loop control
variable was never desirable -- it was just an implementation detail
leaking out. (And that's different from leaking a regular for-loop's
control variable -- since we have 'break' (and 'else') there are some
legitimate use cases. But comprehensions try to be expressions, and here
the side effect is at best useless and at worst a nasty surprise.)


> > and who are familiar with the difference in implementation
> > of comprehensions (though not generator expressions) in Python 2 vs. 3.
>
> That I pretty much am, I think.  In Python 2, comprehensions (the fish)
> were, at least in effect, expanded in-line to a normal for loop.
> Generator expressions (the fowls) were different.  They were, and still
> are, expanded into a temporary generator function whose return value is
> dropped back into the original namespace.  Python 3 turned
> comprehensions (with 2 news varieties thereof) into fowls also,
> temporary functions whose return value is dropped back in the original
> namespace.  The result is that a list comprehension is equivalent to
> list(generator_ expression), even though, for efficiency, it is not
> implemented that way.  (To me, this unification is more a benefit than
> name hiding.)
>

Right, and this consistency convinced me that the change was worth it. I
just really like to be able to say "[... for ...]" is equivalent to
"list(... for ...)", and similar for set and dict.


> (b) proposes to add extra hidden code in and around the temporary
> function to partly undo the isolation.


But it just adds a nonlocal declaration. There's always some hidden code
('def' and 'return' at the very least).


> list comprehensions would no
> longer be equivalent to list(generator_expression), unless
> generator_expressions got the same treatment, in which case they would
> no longer be equivalent to calling the obvious generator function.
> Breaking either equivalence might break someone's code.
>

Ah, there's the rub! I should probably apologize for not clarifying my
terminology more. In the context of PEP 572, when I say "comprehensions" I
include generators! PEP 572 states this explicitly (
https://github.com/python/peps/blame/master/pep-0572.rst#L201-L202).

Certainly PEP 572 intends to add that implicit nonlocal to both
comprehensions and generator expressions. (I just got really tired of
writing that phrase over and over, and at some point I forgot that this is
only a parenthetical remark added in the PEP's latest revision, and not
conventional terminology -- alas. :-)

Part (b) of PEP 572 does several things of things to *retain* consistency:

- The target of := lives in the same scope regardless of whether it occurs
in a comprehension, a generator expression, or just in some other
expression.

- When it occurs in a comprehension or generator expression, the scope is
the same regardless of whether it occurs in the "outermost iterable" or not.

If we didn't have (b) the target would live in the comprehension/genexpr
scope if it occurred in a comprehension/genexp but outside its "outermost
iterable", and in the surrounding scope otherwise.


> ---
>
> How loop variables might be isolated without a nested scope: After a
> comprehension is parsed, so that names become strings, rename the loop
> variables to something otherwise illegal.  For instance, i could become
> '<i>', just as lambda becomes '<lambda>' as the name of the resulting
> function.  Expand the comprehension as in Python 2, except for deleting
> the loop names along with the temporary result name.
>
> Assignment expressions within a comprehension would become assignment
> expressions within the for loop expansion and would automatically add or
> replace values in the namespace containing the comprehension.  In other
> words, I am suggesting that if we want name expressions in
> comprehensions to act as they would in Python 2, then we should consider
> reverting to an altered version of the Python 2 expansion.
>

Possibly this is based on a misunderstanding of my use of "comprehensions".
Also, since your trick can only be used for list/set/dict comprehensions,
but not for generator expressions (at least I assume you don't want it
there) it would actually *reduce* consistency between list/set/dict
comprehensions and generator expressions.


> ---
>
> In any case, I think (b) should be a separate PEP linked to a PEP for
> (a).  The decision for (a) could be reject (making (b) moot), accept
> with (b), or accept unconditionally (but still consider (b)).
>

For me personally, (b) makes the PEP more consistent, so I'm not in favor
of breaking up the PEP. But we can certainly break up the discussion --
that's why I started using the labels (a) and (b).

----------
[1] Sometimes it's an implicit global instead of an implicit nonlocal --
when there's already a global for the same variable in the target scope.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180626/04dfae77/attachment.html>