simple string parsing ?

[Alex Martelli]

TAG <tonino.greco at gmail.com> wrote:

> WOW - I never thought tokenize was that simple :)

It didn't use to be all that simple when it was callback-based, but
since the generate_tokens function was put into it I think it's become
so.  You do need a list comprehension or something over the iterator
which generate_tokens return, and to wrap a readling function around the
string you're tokenizing to pass it (must return the string the first
time, and '' the second time it's called), but that's tolerable IMHO.

((Of course, you ARE restricted to what Python considers 'tokens' so you
may need some postprocessing if you need a slightly different notion of
tokens))

The new iterator protocol has allowed interface simplifications such as
this one and the equally empowering os.walk (iterator based) vs
os.path.walk (callbacl based), which I think is quite a good sign said
protocol is good!-)


Alex