turning callback into generator

[Peter Otten]

Bengt Richter wrote:

> Not that familiar with python threads and queue, but here's a start:
> 
>  >>> import threading
>  >>> import Queue
>  >>> q = Queue.Queue(3) # ridiculously short

But already too long if the items yielded are somehow interdependant.

>  >>>
>  >>> def producer(n, cb):
>  ...     for i in xrange(n): cb(i)
>  ...
>  >>> def product_generator(p, *a, **kw):
>  ...     tpq = threading.Thread(target=p, args=a, kwargs=kw)
>  ...     tpq.start()
>  ...     try:
>  ...         while True: yield q.get(True,5)
>  ...     except Queue.Empty:
>  ...         print 'No data for 5 seconds'
>  ...
>  >>> for item in product_generator(producer, 8, q.put): print item
>  ...
>  0
>  1
>  2
>  3
>  4
>  5
>  6
>  7
>  No data for 5 seconds
>  >>>
> 
> Not tested beyond what you see ;-)

The problem is _not_ to signal that there are no more items to be expected -
you can easily do that by wrapping the producer to put a special object
into the queue when it's done and test for that in the product_generator().
This would avoid the 5-second jet lag. It's a bit harder to avoid a
dangling thread when you have code like

for item in product_generator(...): 
   if pred(item): break

You could attack that with a long timeout for Queue.put(), too, but I'd
rather signal the generator that we're done (via a second Queue, maybe).
Unfortunately this imposes changes on the client code, thus defeating the
original goal of making generators and visitors/callbacks freely
interchangeable.

Peter