Odd name shadowing in comprehension

[Steve D'Aprano]

On Sun, 23 Oct 2016 11:43 am, Terry Reedy wrote:

> On 10/22/2016 7:57 PM, Chris Angelico wrote:
>> This surprised me.
>>
>> Python 3.4.2 (default, Oct  8 2014, 10:45:20)
>> [GCC 4.9.1] on linux
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> y=6
>>>>> [(x,y) for x in range(y) for y in range(3)]
>> [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2,
>> 2), (3, 0), (3, 1), (3, 2), (4, 0), (4, 1), (4, 2), (5, 0), (5, 1),
>> (5, 2)]
>>>>> [(x,y) for x in range(3) for z in range(y) for y in range(3)]
>> Traceback (most recent call last):
>>   File "<stdin>", line 1, in <module>
>>   File "<stdin>", line 1, in <listcomp>
>> UnboundLocalError: local variable 'y' referenced before assignment
>>
>> Normally, a comprehension is described as being equivalent to an
>> unrolled loop, inside a nested function. That would be like this:
>>
>> def temp():
>>     ret = []
>>     for x in range(y):
>>         for y in range(3):
>>             ret.append((x,y))
>>     return ret
>> temp()
> 
> This would make the first example fail, which would not be nice.
> 
> 
>> But it seems that the first iterator (and only that one) is evaluated
>> in the parent context:
> 
> Because the first iterator *can* always be evaluated.

I don't understand what you mean by that. If I take you literally, it is
obviously not true:

py> [x for x in garglebarblewarble]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'garglebarblewarble' is not defined

but I'm sure you know that, so I don't understand what you mean by "always".


According to the normal Python scoping rules[1], variables only come from a
single scope at a time. Lua has different rules: translating into Python,
Lua functions work like this:

x = 'global'
def foo():
    print x  # here, x will be the global x
    x = 'local'
    print x  # but now it is the local x

and foo() will print "global" then "local". But according to Python's
scoping rules, foo must raise NameError, specifically 

UnboundLocalError: local variable 'x' referenced before assignment

So it seems strange that a little bit of Lua's behaviour has crept into list
comprehensions. I doubt that's intentional.


There's definitely something strange going on. Compare the what happens when
the semi-global variable is in the first loop iterable versus the second
loop iterable. In this first example, y refers to both the global and the
local, yet strangely there's no error:


py> y = 999
py> [(y, z, x) for x in (1, y) for z in (10, 20) for y in (100,)]
[(100, 10, 1), (100, 20, 1), (100, 10, 999), (100, 20, 999)]


but if we move the reference to y into the second loop, the usual rule about
undefined local variables is used:

py> [(y, z, x) for x in (1, 2) for z in (10, y) for y in (100,)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
UnboundLocalError: local variable 'y' referenced before assignment


Of course there's no problem with accessing globals in the second loop, so
long as the name doesn't clash with a local:

py> Y = 999
py> [(y, z, x) for x in (1, 2) for z in (10, Y) for y in (100,)]
[(100, 10, 1), (100, 999, 1), (100, 10, 2), (100, 999, 2)]





[1] Function declarations are *slightly* different, so we can write this:

    def func(a, b=b)

to define a parameter (local variable) "b" that takes its default value from
b in the surrounding scope. But that's a declaration, not an expression.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.