Callbacks to generators

[Dave Benjamin]

Is there a straightforward way to create a generator from a function that
takes a callback? For instance, I have a function called "process":

def process(text, on_token):
    ...
    
For each token that "process" finds in "text", it creates a token object,
"token", and calls "on_token(token)".

Now, suppose I wanted to create a generator based on this function. I tried
to do the following:

def process_iter(text):
    def on_token(token):
        yield token
    process(text, on_token)

However, rather than create a generator as I had hoped, this function
doesn't return anything. I guess it creates a bunch of singleton generators,
one per token, and throws them away. In any case, is there a way to do what
I am trying to do here without resorting to threads?

The reason I am trying to do this is that I'd still like to be able to use
"process" in code for Python 2.1, which does not support generators. I'd
like to avoid implementing the entire thing twice or basing the callback
version on the generator version.

Thanks,
Dave

-- 
.:[ dave benjamin: ramen/[sp00] -:- spoomusic.com -:- ramenfest.com ]:.
: i'm half drunk on babble you transmit - soul coughing / true dreams :