Why doesn't eval of generator expression work with locals?

[Gabriel Genellina]

En Fri, 30 Jan 2009 07:08:29 -0200, Hendrik van Rooyen  
<mail at microcorp.co.za> escribió:

> "Gabriel Genellina" <gagsl-py2 at yahoo.com.ar> wrote:
>
> Of course this is clearly stated in the Language Reference "Variables  
> used
> in the generator expression are evaluated lazily in a separate scope when
> the next() method is called for the generator object (in the same fashion
> as for normal generators). However, the in expression of the leftmost for
> clause is immediately evaluated in the current scope..." -- but this
> behaviour is still surprising and not obvious to me. ("not obvious" means
> that things could have been different, choosing this was a design
> decision).
>
> I am not so sure that it could have been done differently -
> I see it something like this:  (going back to almost your
> original example, and reversing the position of globals
> and locals to make it shorter)
>
>>>> def foo(things):
>  for thing in things:
>   yield thing()    #it is obvious this is in the local scope of foo
>
>>>> boo = foo([locals,globals])
>>>> boo.next()
> {'thing': <built-in function locals>, 'things': [<built-in function  
> locals>,
> <built-in function globals>]}
>>>>
> and so it is, when you feed it the locals function
> [...]
> Now I don't think that you could really do it differently -
> the right hand side of the generator expression is exactly
> like my passed argument "things", in all cases as far as
> I can see, and this means that the right hand side is
> evaluated when it is "passed", and the left hand side
> is whatever is done in the "for thing in things:" loop.
> All the generator expression does is that it saves you
> the trouble of defining the function - it kind of does it
> for you, and calls it, and returns the generator object,
> and throws the function away, all in one hit. (this is not
> necessarily the real mechanism, but the effect is exactly
> as if it were)

Yes, but this is not the only alternative. You *decided* that foo and bar  
will take one argument - this means that it uses early binding and it is  
evaluated when the generator expression is created. This is a design  
decision, and it could have been different.
The left-part of the generator expression (in the function analogy, the  
"yield" expression) is late bound - it is completely evaluated at every  
iteration, using whatever values are currently bound to external (free)  
variables. The same *could* happen with the right part too -- although  
this is not what was decided.

Consider this expression: g = (x+A for x in L for y in M). This is  
currently expanded more or less like this:

def foo(L):
   for x in L:
     for y in M:
       yield x+A
g = foo(iter(L))

(as your example above) Note that L has a special status -- it's the only  
expression evaluated at the time g is defined. It *could* have been like  
this:

def foo()
   for x in L:
     for y in M:
       yield x+A
g = foo()

or even like this:

def foo(L, M, A):
   for x in L:
     for y in M:
       yield x+A
g = foo(iter(L), iter(M), A)

In particular, I like the 2nd (all late binding). Seems this topic was  
discussed many times [1] when PEP289 [2] was proposed, and "practicality  
beats purity".

[1] http://mail.python.org/pipermail/python-dev/2004-April/date.html
[2] http://www.python.org/dev/peps/pep-0289/

-- 
Gabriel Genellina