Duncan Booth wrote: > Why would you return a token rather than throwing an exception? Tokenizers have lots of uses. Colorizing text in an editor, for example. We've got a MALFORMED_NUMBER when you type '0x'. We've got an INTEGER when we get your next keystroke (probably).