Callbacks to generators

[Dominic Fox]

It would be nice if something like this *could* work - you could instantly
and effortlessly transform a SAX parser into a .Net-style XmlReader. Alas,
it cannot...

"yield" in a function causes the function to return a generator object to
the caller - in this case, the caller of on_token, not the caller of
process_iter. The inner function "on_token" in "process_iter" returns a
generator when it is called by "process", and "process" promptly throws this
generator away. The value returned by "process_iter" is just the return
value from "process".

It might be possible to do something with call/cc, but of course Python
doesn't have that.

Dominic

-----Original Message-----
From: Humpty Dumpty [mailto:oliver.schoenborn at utoronto.ca] 
Sent: 08 June 2004 13:07
To: python-list at python.org
Subject: Re: Callbacks to generators

> def process(text, on_token):
>     ...
>
> For each token that "process" finds in "text", it creates a token object,
> "token", and calls "on_token(token)".
>
> Now, suppose I wanted to create a generator based on this function. I
tried
> to do the following:
>
> def process_iter(text):
>     def on_token(token):
>         yield token
>     process(text, on_token)
>
> However, rather than create a generator as I had hoped, this function
> doesn't return anything. I guess it creates a bunch of singleton
generators,
> one per token, and throws them away. In any case, is there a way to do
what
> I am trying to do here without resorting to threads?

Something weird here. I hven't used generators much, but seems to me that:

1) you maybe don't need them:

 def process_iter(text):
     def on_token(token):
         return token
     process(text, on_token)

2) you need some sort of while loop so the fnction doesn't return after the
first yield:

 def process_iter(text):
     def on_token(token):
         while 1:
            yield token
     process(text, on_token)

I'm curious to understand if neither of those are true, why...

Oliver