[Python-ideas] A comprehension scope issue in PEP 572

[Terry Reedy]

On 5/7/2018 1:38 PM, Guido van Rossum wrote:
> I am convinced by Tim's motivation. I hadn't thought of this use case 
> before -- I had mostly thought "local scope in a comprehension or 
> generator expression is the locals of the synthetic function". But Tim's 
> reasoning feels right.
> 
> The only solution that makes sense to me is Steven's (2). (1) is what 
> the PEP currently says and what Tim doesn't want; (3) has no precedent 
> (function defaults don't really work like this) and just gets my hackles 
> all up. (I can't even tell if Steven meant it as a serious proposal.)
> 
> So let's see if we can change PEP 572 so that := inside a comprehension 
> or generator expression al ways assigns to a variable in the containing 
> scope.
> 
> It may be inconsistent with the scope of the loop control variable, but 
> it's consistent with uses of := outside comprehensions:
> 
>    [x := 0]
>    [x := i for i in range(1)]
> 
> both have the side effect of setting x to zero. I like that.

If I am understanding correctly, this would also let one *intentionally 
'leak' (export) the last value of the loop variable when wanted.

[math.log(xlast:=x) for x in it if x > 0]
print(xlast)

> There's one corner case (again) -- class scopes. If the containing scope 
> is a function, everything's fine, we can use the existing closure 
> machinery. If the containing scope is global, everything's fine too, we 
> can treat it as a global. But if the containing scope is a class, we 
> can't easily extend the current machinery. But this breakage is similar 
> to the existing breakage with comprehensions in class scope that 
> reference class variables:
> 
>    class C:
>        hosts = ['boring', 'haring', 'tering']
>        full_hosts = [host + suffix for suffix in ('.cwi.nl 
> <http://cwi.nl>', '.com') for host in hosts]
> 
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "<stdin>", line 3, in C
>    File "<stdin>", line 3, in <listcomp>
> NameError: name 'hosts' is not defined

This is a special case of the fact that no function called in class 
scope can access class variables, even if defined in the class scope.

 >>> class C:
	x = 0
	def f():
		return x
	z = f()
	
Traceback (most recent call last):
   File "<pyshell#5>", line 1, in <module>
     class C:
   File "<pyshell#5>", line 5, in C
     z = f()
   File "<pyshell#5>", line 4, in f
     return x
NameError: name 'x' is not defined

I would find it strange if only functions defined by a comprehension 
were given new class scope access.

> PS1. The various proposals that add random extra keywords to the syntax 
> (like 'for nonlocal i') don't appeal to me at all.
> 
> PS2. IIRC the reason we gave loop control variables their own scope was 
> the poor imagination of many people when it comes to choosing loop 
> control variable names. We had seen just too many examples of
> 
>    for x in something:
>        ...lots of code using x...
>        blah blah [x+1 for x in something else]
>        ...some more code using x, broken...
> 
> It's true that this can also happen with a for-loop statement nested 
> inside the outer loop (and it does) but the case of a comprehension was 
> easier to miss. I've never looked back.
> 
> PS3. Comprehensions and generator expressions should be interchangeable. 
> They just look too similar to have different semantics (and the possibly 
> delayed execution of generator expressions is not an issue -- it's rare, 
> and closure semantics make it work just fine).

To me, this is the prime justification for the 3.0 comprehension change. 
  I currently see a comprehension as a specialized generator expression. 
  A generator expression generalizes math set builder notation. If left 
'raw', the implied function yields the values generated (what else could 
it do?).  If a collection type is indicated by the fences and expression 
form, values are instead added to an anonymous instance thereof.

-- 
Terry Jan Reedy