Basic tokenizer

[Dale Strickland-Clak]

I'm looking for a simple string tokenizer. The Python tokenize module is 
too specific and is geared to Python syntax.

I just want something that can be setup to read the basic stuff like 
. identifiers 
. integers (and/or reals)
. strings
. special characters such as operators and brackets.

Is anyone aware of such a module?

Thanks.

-- 
Dale Strickland-Clark
Riverhall Systems Ltd, www.riverhall.co.uk