Parsing variables out of python lines

[Bengt Richter]

On Mon, 07 Oct 2002 05:43:54 GMT, Mark <Hobbes2176 at yahoo.com> wrote:

>I hate finding answer quickly after posting.  Anyway, here is code that will
>solve my problem:
>
>
>import tokenize
>import StringIO
>
>foo = [1,2,3]
>x = "for i in foo:"
>
>#
>#The StringIO trick turns a string into a read'able object (or so it seems)
>#
>t = tokenize.generate_tokens(StringIO.StringIO(x).readline)
>
>for junk, item, rest1, rest2, rest3 in t:
>    if item in vars():
>        print eval(item),
>    else:
>        print item,
>print
>
>With output:
>==> for i in [1, 2, 3] :
>
>Anyone know how to clean up the tuple assignment (I only want item, the rest 
>could go to /dev/null if they want).
>
Not sure what you're doing in your larger context, but extracting the second item
from each tuple is easy with a list comprehension (see below). If the sequence is
really huge, you could wrap tokenize in a generator to return the selected tuple
element. (See further below).
 
(BTW, if you know that item in vars() is true, why not just look it up with vars()[item]?)

 >>> import tokenize
 >>> import StringIO
 >>> foo = [1,2,3]
 >>> x = "for i in foo:"
 >>> items = [tup[1] for tup in tokenize.generate_tokens(StringIO.StringIO(x).readline)]
 >>> print ' '.join([(z in vars() and [`vars()[z]`] or [z])[0] for z in items])
 for i in [1, 2, 3] :

Though you might want to do `` to all the output so you can distinguish better (and see the ''):
>>> print ' '.join([`(z in vars() and [vars()[z]] or [z])[0]` for z in items])
'for' 'i' 'in' [1, 2, 3] ':' ''

or if you don't mind the brackets and commas, just let the list be repr'd
>>> print [(z in vars() and [vars()[z]] or [z])[0] for z in items]
['for', 'i', 'in', [1, 2, 3], ':', '']

The generator lets you put "in tokitems(x,1)" in place of "in items" above:

 >>> from __future__ import generators
 >>> def tokitems(s, i):
 ...     for tup in tokenize.generate_tokens(StringIO.StringIO(s).readline):
 ...         yield tup[i]
 ...
 >>> print ' '.join([`(z in vars() and [vars()[z]] or [z])[0]` for z in tokitems(x,1)])
 'for' 'i' 'in' [1, 2, 3] ':' ''

Or, less obscure and avoiding the ugly substitute for (x?y:z),

 >>> for item in tokitems(x,1):
 ...     try:
 ...         print `vars()[item]`,
 ...     except KeyError:
 ...         print `item`,
 ...
 'for' 'i' 'in' [1, 2, 3] ':' ''

(I can't add the final dedented bare "print" interactively on the next line for
some reason. If it's waiting for a blank line, why can't it wait for that blank
line and decide *then* whether I've done badly with the dedent? It would make
for cleaner example-typing. I can get around it by putting everything indented
under an "if 1:" but IMO that's a warty restriction.)

Regards,
Bengt Richter