[Python-Dev] Seeming unintended difference between list comprehensions and generator expressions...

[Josiah Carlson]

On Fri, Feb 20, 2009 at 3:14 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Josiah Carlson wrote:
>> The behavior of 3.0 WRT list comprehensions behaving the same way as
>> generator expressions is expected, but why generator expressions
>> (generally) don't keep a reference to the class scope during execution
>> seems to be unintended.
>
> It's intended. While arguably not ideal, it would require some pretty
> major changes to the lexical scoping rules to make them behave any
> differently.
>
> The translation of (i*i for i in x) is conceptually along the lines of:
>
>  def _ge(arg):
>    for i in arg:
>      yield i*i
>
>  <expr_value> = _ge(x)
>
> Similarly, a 3.x list comprehension [i*i for i in x] is very roughly
> translated as:
>
>  def _lc(arg):
>    result = []
>    for i in arg:
>      result.append(i*i)
>    return result
>
>  <expr_value> = _lc(x)

I was under the impression that in 3.x, it was equivalent to
list(<genexp>) .  Which would explain the difference between 2.6 and
3.0.

> Like any function scope inside a class namespace, the body of a genexp
> (and, in 3.x, comprehension) doesn't have direct access to the class
> namespace because classes don't play any part in the lexical scoping rules.

Indeed, though I had thought (if only briefly ;) ) that when executing
non-definitions in the class body, it would behave similar to the a
more or less equivalent function-based class factory

>>> def make_class(*bases):
...     def make_class(fcn):
...             dict = fcn()
...             return type(object)(fcn.__name__, bases, dict)
...     return make_class
...
>>> @make_class(object)
... def foo():
...     x = {}
...     x.update((i, x.get(i, None)) for i in xrange(10))
...     return locals()
...
>>> foo
<class '__main__.foo'>
>>> foo.x
{0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7:
None, 8: None, 9: None}
>>>

But I was wrong ;)

Thank you for the help :)
 - Josiah

> Basically, if a generator or 3.x comprehension needs access to a value
> from a containing class scope anywhere other than the outermost
> iterator, then it needs to be put into a temporary function and given
> the extra value as an argument:
>
> .>> class C:
> ...   x = {}
> ...   def _init_constants(d, itr):
> ...     d.update((i, d.get(i, None)) for i in itr)
> ...   _init_constants(x, range(10))
> ...   del _init_constants
> ...
> .>> C.x
> {0: None, 1: None, 2: None, 3: None, 4: None, 5: None, 6: None, 7: None,
> 8: None, 9: None}
>
> (in this toy case, of course, it would be simpler to make the temporary
> function create and return the constants dictionary, but the above
> approach with multiple arguments being passed in applies more generally
> when you need to access multiple existing values from the class scope)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>