Turning f(callback) into a generator

[Peter Otten]

Jeff Epler wrote:

> On Wed, Dec 03, 2003 at 06:07:08PM -0800, Jimmy Retzlaff wrote:
>> My solution used threads. I'd love to know if someone comes up with a
>> solution not involving threads. Here's a simple example of my
>> thread-based approach:
> [...]
> 
> In standard Python, this is the only approach that can work.  It's
> possible that Stackless Python might provide a solution that doesn't
> involve OS threads, but I don't have any actual experience with
> Stackless.
> 
> One problem with your code is the use of a sentinel value.  Another
> poster suggested using multiple queues and a timeout to return
> exceptions or end-of-queue.  Instead, you should use a queue but insert
> easily distinguishable items in it.  For instance:
>     None: end of iterator, raise StopIteration
>     (0, a, b): exception in f, raise a, None, b (re-raise exception)
>     (1, blah): callback got blah, return it from next()
> 
> 
> import sys, Queue, threading
> 
> class CallbackGenerator:
>     def __init__(self, func, pre=(), post=()):
>         self.func = func
>         if type(pre) is not tuple: pre = (pre,)
>         if type(post) is not tuple: post = (post,)
>         self.args = pre + (self.cb,) + post
>         self.queue = Queue.Queue(maxsize=1)
>         self.thread = threading.Thread(target=self.go)
>         self.thread.start()
> 
>     def __iter__(self): return self
> 
>     def next(self):
>         item = self.queue.get()
>         if item is None:
>             raise StopIteration
>         if item[0] == 0:
>             raise item[1], None, item[2]
>         else:
>             return item[1]
> 
>     def go(self):
>         try:
>             self.func(*self.args)
>         except:
>             info = sys.exc_info()
>             self.queue.put((0, info[1], info[2]))
>         else:
>             self.queue.put(None)
> 
>     def cb(self, *args):
>         self.queue.put((1, args))
> 
> # Example 1: os.path.walk -> generator
> import os
> for i in CallbackGenerator(os.path.walk, "/tmp", None):
>     print i
> 
> # Example 2: Prints 1, 2 then shows a traceback
> def f(cb):
>     cb(1)
>     cb(2)
>     1/0
> 
> for i in CallbackGenerator(f):
>     print i

Your code looks far more concise than mine.
However, it suffers from the same problem:

# nasty example
maxRepeat = 0
for i in CallbackGenerator(os.path.walk, "/tmp", None):
    if maxRepeat == 0:
        break
    print i
    maxRepeat -= 1

How do you handle the case where the generator is not run to exhaustion?

Peter