How to in Python

[MartinRinehart at gmail.com]

Gabriel Genellina wrote:
> Do you have to validate input based on that grammar?

I've built a standalone tokenizer. It returns an array of Token
objects. These include tokens such as UNCLOSED_QUOTE and
MALFORMED_NUMBER ('1E' not followed by sign or digit, for instance).

You could use this in a code editor to colorize. You could use it in a
documentation writer to emit source in HTML. You could use the tokens
in a parser.