turning callback into generator

[Wai Yip Tung]

On Tue, 07 Sep 2004 10:15:55 +0200, Peter Otten <__peter__ at web.de> wrote:

> Bengt Richter wrote:
>
>> Not that familiar with python threads and queue, but here's a start:
>>
>>  >>> import threading
>>  >>> import Queue
>>  >>> q = Queue.Queue(3) # ridiculously short
>
> But already too long if the items yielded are somehow interdependant.
>
>>  >>>
>>  >>> def producer(n, cb):
>>  ...     for i in xrange(n): cb(i)
>>  ...
>>  >>> def product_generator(p, *a, **kw):
>>  ...     tpq = threading.Thread(target=p, args=a, kwargs=kw)
>>  ...     tpq.start()
>>  ...     try:
>>  ...         while True: yield q.get(True,5)
>>  ...     except Queue.Empty:
>>  ...         print 'No data for 5 seconds'
>>  ...
>>  >>> for item in product_generator(producer, 8, q.put): print item
>>  ...
>>  0
>>  1
>>  2
>>  3
>>  4
>>  5
>>  6
>>  7
>>  No data for 5 seconds
>>  >>>
>>
>> Not tested beyond what you see ;-)
>
> The problem is _not_ to signal that there are no more items to be  
> expected -
> you can easily do that by wrapping the producer to put a special object
> into the queue when it's done and test for that in the  
> product_generator().
> This would avoid the 5-second jet lag. It's a bit harder to avoid a
> dangling thread when you have code like
>
> for item in product_generator(...):
>    if pred(item): break
>
> You could attack that with a long timeout for Queue.put(), too, but I'd
> rather signal the generator that we're done (via a second Queue, maybe).
> Unfortunately this imposes changes on the client code, thus defeating the
> original goal of making generators and visitors/callbacks freely
> interchangeable.
>
> Peter
>

Thanks for the pointers. Actually I'm looking for a solution that do not  
use thread or to buffer all result in memory since one of the great thing  
about generator is that it can achieve the process without using those  
resource intensive techniques.

About the dangling thread, perhaps the destructor method __del__() can  
help (http://www.python.org/dev/doc/devel/ref/customization.html). It is  
really nasty to end up with dangling threads.

Wai Yip