generic tokenizer

Angus Mackay yeah at right.com
Tue Aug 31 14:30:26 EDT 2004


I remember python having a generic tokenizer in the library. all I want 
is to set a list of token seperators and then read tokens out of a 
stream, the token seperators should be returned as themselves.

is there anything like this?

cheers, Angus.



More information about the Python-list mailing list