generic tokenizer

[Miki Tebeka]

Hello Angus,

> I remember python having a generic tokenizer in the library. all I want 
> is to set a list of token seperators and then read tokens out of a 
> stream, the token seperators should be returned as themselves.
> 
> is there anything like this?
There are external lex/yacc like packages (like PLY
http://systems.cs.uchicago.edu/ply/). But maybe you're talking about shlex
(http://www.python.org/dev/doc/devel/lib/module-shlex.html)

HTH.
--
------------------------------------------------------------------------
Miki Tebeka <miki.tebeka at zoran.com>
http://tebeka.spymac.net
The only difference between children and adults is the price of the toys