Turning f(callback) into a generator

[Jimmy Retzlaff]

Peter Otten wrote:

[Nice generalization of a thread-based generator, thanks]

> The question that has come up writing the above, how you would handle
> exceptions in the thread? I put them in the queue and reraise them in
the
> main thread, but I'm not feeling very confident about it...

Perhaps there should be a second queue for an exception and the
sentinel. Then the main loop could look something like this (untested):

    while terminationQueue.empty() or not queue.empty():
        # The timeout is needed because we may have arrived here after
        # the last value was placed in the queue but before the sentinel
        # or an exception was placed in the terminationQueue.
        yield queue.get(timeout=0.1)

    terminationValue = terminationQueue.get()
    if terminationValue is not sentinel:
         raise terminationValue

In this case the launcher function would put the sentinel in the
terminationQueue instead of the data queue. The idea of the loop
condition above is to yield everything that came through before the
exception/sentinel, then re-raise any exception if that was the reason
for terminating. This would hopefully help clarify where the exception
occurred (i.e., on which element in the iteration).

The separate queue also eliminates the problem of someone somehow
finding a way of injecting the sentinel into the data stream. But this
does create the need for the timeout (which doesn't work in 2.2). I
guess it's a judgment call as to whether the sentinel should be placed
in one queue or the other (or both).

Your generalization could make a nice Python Cookbook recipe if you feel
like putting it together. At least two of us have needed this sort of
thing.

Jimmy