python LEX

[Michael Spencer]

Mike Meyer wrote:

...
> 
> Trouble is, he wants a lexer, not a parser. The lexer in the sources
> is an ad-hoc package, contained in Parser/tokenizer.c.
> 
>    <mike

There is also tokenize.py in the standard library, which might be an easier 
reference

"""
...
It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators
...
"""

Michael