Simple eval

[greg]

Tor Erik Sønvisen wrote:
> Comments, speedups, improvements in general, etc are appreciated.

You're doing a lot of repeated indexing of token[0]
and token[1] in your elif branches. You might gain some
speed by fetching these into locals before entering the
elif chain.

Also you could try ordering the branches so that the
most frequent cases come first. Probably strings and
numbers first, then the various kinds of bracket.
This would also give you a chance to avoid pulling out
token[1] until you need it.

token[1].startswith('u'): It's probably faster to
use an index to get the first character, if you know
that the string is not empty.

Importing the names from tokenize that you use repeatedly
should save some time, too.

Putting all these together, it would look something
like

from tokenize import STRING, NUMBER

   def atom(next, token):
     token0 = token[0]
     if token0 == STRING
       ...
     elif token0 == NUMBER:
       ...
     elif token0[0] == 'u':
       ...
     else:
       token1 = token[1]
       if token1 in KEYWORDS:
         ...
       elif token1 == '(':
         ...
       elif token1 == '[':
         ...
       elif token1 == '{':
         ...

If you were willing to indulge in some default-argument abuse, you
could also do

   def atom(next, token, STRING = tokenize.STRING, NUMBER = tokenize.NUMBER):
     ...

A more disciplined way would be to wrap it in a closure:

   def make_atom():
     from tokenize import STRING, NUMBER
     def atom(next, token):
       ...
     return atom

   atom = make_atom()

--
Greg