Callbacks to generators

[Humpty Dumpty]

> def process(text, on_token):
>     ...
>
> For each token that "process" finds in "text", it creates a token object,
> "token", and calls "on_token(token)".
>
> Now, suppose I wanted to create a generator based on this function. I
tried
> to do the following:
>
> def process_iter(text):
>     def on_token(token):
>         yield token
>     process(text, on_token)
>
> However, rather than create a generator as I had hoped, this function
> doesn't return anything. I guess it creates a bunch of singleton
generators,
> one per token, and throws them away. In any case, is there a way to do
what
> I am trying to do here without resorting to threads?

Something weird here. I hven't used generators much, but seems to me that:

1) you maybe don't need them:

 def process_iter(text):
     def on_token(token):
         return token
     process(text, on_token)

2) you need some sort of while loop so the fnction doesn't return after the
first yield:

 def process_iter(text):
     def on_token(token):
         while 1:
            yield token
     process(text, on_token)

I'm curious to understand if neither of those are true, why...

Oliver