Tokenizer for python?

[Nathan Harmston]

Hi,

>From your description I think something like PyParsing might be useful to you.

http://pyparsing.wikispaces.com/

If not then I guess I m not understanding your question right.

Hope this helps

Nathan

On 21/08/07, Robert Dailey <rcdailey at gmail.com> wrote:
> Hi,
>
> I am looking for a sort of "tokenizer" for python. I've taken a look at the
> tokenize module, but that seems to parse python code from what I read. I
> want a tokenizer that works a little like boost::tokenizer, however for
> python. Basically I want to be able to pass in an arbitrary string (or line
> from readline()) and specify tokens that cause the string to be separated
> into parts, much like the regular expression split() method (I think that's
> the name of it). Is there anything that already exists that does this, or do
> I need to implement it myself with regular expressions?
>
> Thanks.
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>