Producer-consumer threading problem

[Larry Bates]

George Sakkis wrote:
> On Jun 10, 11:47 pm, Larry Bates <larry.ba... at websafe.com`> wrote:
>> I had a little trouble understanding what exact problem it is that you are
>> trying to solve but I'm pretty sure that you can do it with one of two methods:
> 
> Ok, let me try again with a different example: I want to do what can
> be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
> (see example at http://docs.python.org/lib/QueueObjects.html), but
> instead of  having to first put all items and then wait until all are
> done, get each item as soon as it is done.
> 
>> 1) Write the producer as a generator using yield method that yields a result
>> every time it is called (something like os.walk does).  I guess you could yield
>> None if there wasn't anything to consume to prevent blocking.
> 
> Actually the way items are generated is not part of the problem; it
> can be abstracted away as an arbitrary iterable input. As with all
> iterables, "there are no more items" is communicated simply by a
> StopIteration.
> 
>> 2) Usw somethink like Twisted insted that uses callbacks instead to handle
>> multiple asynchronous calls to produce.  You could have callbacks that don't do
>> anything if there is nothing to consume (sort of null objects I guess).
> 
> Twisted is interesting and very powerful but requires a different way
> of thinking about the problem and designing a solution. More to the
> point, callbacks often provide a less flexible and simple API than an
> iterator that yields results (consumed items). For example, say that
> you want to store the results to a dictionary. Using callbacks, you
> would have to explicitly synchronize each access to the dictionary
> since they may fire independently. OTOH an iterator by definition
> yields items sequentially, so the client doesn't have to bother with
> synchronization. Note that with "client" I mean the user of an API/
> framework/library; the implementation of the library itself may of
> course use callbacks under the hood (e.g. to put incoming results to a
> Queue) but expose the API as a simple iterator.
> 
> George

If you use a queue and the producer/collector are running in different threads 
you don't have to wait for "to first put all items and then wait until all are
done".  Producer can push items on the queue and while the collector 
asynchronously pops them off.

I'm virtually certain that I read on this forum that dictionary access is atomic 
if that helps.

-Larry