List comprehension - NameError: name '_[1]' is not defined ?

[mario ruggier]

On Jan 15, 2:02 pm, Peter Otten <__pete... at web.de> wrote:
> mario ruggier wrote:
> > Hello,
>
> > I would like to evaluate list comprehension expressions, from within
> > which I'd like to call a function. For a first level it works fine but
> > for second level it seems to lose the "_[1]" variable it uses
> > internally to accumulate the results. Some sample code is:
>
> > class GetItemEvaluator(object):
> >     def __init__(self):
> >         self.globals = globals() # some dict (never changes)
> >         self.globals["ts"] = self.ts
> >         self.globals["join"] = "".join
> >         self.locals = {} # changes on each evaluation
> >     def __getitem__(self, expr):
> >         return eval(expr, self.globals, self.locals)
> >     def ts(self, ts, name, value):
> >         self.locals[name] = value
> >         #print ts, name, value, "::::", self.locals, "::::", ts % self
> >         return ts % self
>
> > gie = GetItemEvaluator()
> > gie.locals["inner"] = ("a","b","c","d")
> > print """
> > pre %(join([ts("%s."%(j)+'%(k)s ', 'k', k) for j,k in enumerate
> > (inner)]))s post
> > """ % gie
> > # OK, outputs: pre 0.a 1.b 2.c 3.d  post
>
> > gie = GetItemEvaluator()
> > gie.locals["outer"] = [ ("m","n","o","p"), ("q","r","s","t")]
> > print """
> > pre %(join([ts(
> >   '''inner pre
> >   %(join([ts("%s.%s."%(i, j)+'%(k)s ', 'k', k) for j,k in enumerate
> > (inner)]))s
> >    inner post''',
> >   "inner", inner) # END CALL outer ts()
> >   for i,inner in enumerate(outer)])
> > )s post
> > """ % gie
>
> > The second 2-level comprehension gives:
>
> >   File "scratch/eval_test.py", line 8, in __getitem__
> >     return eval(expr, self.globals, self.locals)
> >   File "<string>", line 4, in <module>
> > NameError: name '_[1]' is not defined
>
> > If the print was to be enable, the last line printed out is:
>
> > 0.3.%(k)s  k p :::: {'outer': [('m', 'n', 'o', 'p'), ('q', 'r', 's',
> > 't')], 'i': 0, 'k': 'p', 'j': 3, '_[1]': ['0.0.m ', '0.1.n ', '0.2.o
> > '], 'inner': ('m', 'n', 'o', 'p')} :::: 0.3.p
>
> > i.e. it has correctly processed the first inner sequence, until the
> > (last) "p" element. But on exit of the last inner ts() call, it seems
> > to lose the '_[1]' on self.locals.
>
> > Any ideas why?
>
> > Note, i'd like that the first parameter to ts() is as independent as
> > possible from teh context in expression context, a sort of independent
> > mini-template. Thus, the i,j enumerate counters would normally not be
> > subbed *within* the comprehension itself, but in a similar way to how
> > k is evaluated, within the call to ts() -- I added them this way here
> > to help follow easier what the execution trail is. Anyhow, within that
> > mini-template, i'd like to embed other expressions for the % operator,
> > and that may of course also be list comprehensions.
>
> I have no idea what you are trying to do. Please reread the Zen of Python ;)
>
> What happens is:
>
> List comprehensions delete the helper variable after completion:
>
> >>> def f(): [i for i in [1]]
> ...
> >>> dis.dis(f)
>
>   1           0 BUILD_LIST               0
>               3 DUP_TOP
>               4 STORE_FAST               0 (_[1])
>               7 LOAD_CONST               1 (1)
>              10 BUILD_LIST               1
>              13 GET_ITER
>         >>   14 FOR_ITER                13 (to 30)
>              17 STORE_FAST               1 (i)
>              20 LOAD_FAST                0 (_[1])
>              23 LOAD_FAST                1 (i)
>              26 LIST_APPEND
>              27 JUMP_ABSOLUTE           14
>         >>   30 DELETE_FAST              0 (_[1])
>              33 POP_TOP
>              34 LOAD_CONST               0 (None)
>              37 RETURN_VALUE
>
> If you manage to run two nested listcomps in the same namespace you get a
> name clash and the inner helper variable overwrites/deletes the outer:
>
> >>> def xeval(x): return eval(x, ns)
> ...
> >>> ns = dict(xeval=xeval)
> >>> xeval("[xeval('[k for k in ()]') for i in (1,)]")
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "<stdin>", line 1, in xeval
>   File "<string>", line 1, in <module>
> NameError: name '_[1]' is not defined
>
> Peter

Ah, brilliant, thanks for the clarification!

To verify if I understood you correctly, I have modified
the ts() method above to:

    def ts(self, ts):
        _ns = self.locals
        self.locals = self.locals.copy()
        print "ts:", ts, "::::", self.locals
        try:
            return ts % self
        finally:
            self.locals = _ns

And, it executes correctly, thus the 2nd output is:

Output 2:
leading
pre 0.0.m 0.1.n 0.2.o 0.3.p post

pre 1.0.q 1.1.r 1.2.s 1.3.t post
 trailing

But, the need to do a copy() will likely kill any potential
optimization gains... so, I will only be forced to rite more readable
code ;-)

Thanks!