List comprehension - NameError: name '_[1]' is not defined ?

[Peter Otten]

mario ruggier wrote:

> Hello,
> 
> I would like to evaluate list comprehension expressions, from within
> which I'd like to call a function. For a first level it works fine but
> for second level it seems to lose the "_[1]" variable it uses
> internally to accumulate the results. Some sample code is:
> 
> class GetItemEvaluator(object):
>     def __init__(self):
>         self.globals = globals() # some dict (never changes)
>         self.globals["ts"] = self.ts
>         self.globals["join"] = "".join
>         self.locals = {} # changes on each evaluation
>     def __getitem__(self, expr):
>         return eval(expr, self.globals, self.locals)
>     def ts(self, ts, name, value):
>         self.locals[name] = value
>         #print ts, name, value, "::::", self.locals, "::::", ts % self
>         return ts % self
> 
> gie = GetItemEvaluator()
> gie.locals["inner"] = ("a","b","c","d")
> print """
> pre %(join([ts("%s."%(j)+'%(k)s ', 'k', k) for j,k in enumerate
> (inner)]))s post
> """ % gie
> # OK, outputs: pre 0.a 1.b 2.c 3.d  post
> 
> gie = GetItemEvaluator()
> gie.locals["outer"] = [ ("m","n","o","p"), ("q","r","s","t")]
> print """
> pre %(join([ts(
>   '''inner pre
>   %(join([ts("%s.%s."%(i, j)+'%(k)s ', 'k', k) for j,k in enumerate
> (inner)]))s
>    inner post''',
>   "inner", inner) # END CALL outer ts()
>   for i,inner in enumerate(outer)])
> )s post
> """ % gie
> 
> The second 2-level comprehension gives:
> 
>   File "scratch/eval_test.py", line 8, in __getitem__
>     return eval(expr, self.globals, self.locals)
>   File "<string>", line 4, in <module>
> NameError: name '_[1]' is not defined
> 
> If the print was to be enable, the last line printed out is:
> 
> 0.3.%(k)s  k p :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r', 's',
> 't')], 'i': 0, 'k': 'p', 'j': 3, '_[1]': ['0.0.m ', '0.1.n ', '0.2.o
> '], 'inner': ('m', 'n', 'o', 'p')} :::: 0.3.p
> 
> i.e. it has correctly processed the first inner sequence, until the
> (last) "p" element. But on exit of the last inner ts() call, it seems
> to lose the '_[1]' on self.locals.
> 
> Any ideas why?
> 
> Note, i'd like that the first parameter to ts() is as independent as
> possible from teh context in expression context, a sort of independent
> mini-template. Thus, the i,j enumerate counters would normally not be
> subbed *within* the comprehension itself, but in a similar way to how
> k is evaluated, within the call to ts() -- I added them this way here
> to help follow easier what the execution trail is. Anyhow, within that
> mini-template, i'd like to embed other expressions for the % operator,
> and that may of course also be list comprehensions.

I have no idea what you are trying to do. Please reread the Zen of Python ;)

What happens is:

List comprehensions delete the helper variable after completion:

>>> def f(): [i for i in [1]]
...
>>> dis.dis(f)
  1           0 BUILD_LIST               0
              3 DUP_TOP
              4 STORE_FAST               0 (_[1])
              7 LOAD_CONST               1 (1)
             10 BUILD_LIST               1
             13 GET_ITER
        >>   14 FOR_ITER                13 (to 30)
             17 STORE_FAST               1 (i)
             20 LOAD_FAST                0 (_[1])
             23 LOAD_FAST                1 (i)
             26 LIST_APPEND
             27 JUMP_ABSOLUTE           14
        >>   30 DELETE_FAST              0 (_[1])
             33 POP_TOP
             34 LOAD_CONST               0 (None)
             37 RETURN_VALUE

If you manage to run two nested listcomps in the same namespace you get a
name clash and the inner helper variable overwrites/deletes the outer:

>>> def xeval(x): return eval(x, ns)
...
>>> ns = dict(xeval=xeval)
>>> xeval("[xeval('[k for k in ()]') for i in (1,)]")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in xeval
  File "<string>", line 1, in <module>
NameError: name '_[1]' is not defined

Peter