Producer-consumer threading problem

[George Sakkis]

On Jun 10, 11:47 pm, Larry Bates <larry.ba... at websafe.com`> wrote:
>
> I had a little trouble understanding what exact problem it is that you are
> trying to solve but I'm pretty sure that you can do it with one of two methods:

Ok, let me try again with a different example: I want to do what can
be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
(see example at http://docs.python.org/lib/QueueObjects.html), but
instead of  having to first put all items and then wait until all are
done, get each item as soon as it is done.

> 1) Write the producer as a generator using yield method that yields a result
> every time it is called (something like os.walk does).  I guess you could yield
> None if there wasn't anything to consume to prevent blocking.

Actually the way items are generated is not part of the problem; it
can be abstracted away as an arbitrary iterable input. As with all
iterables, "there are no more items" is communicated simply by a
StopIteration.

> 2) Usw somethink like Twisted insted that uses callbacks instead to handle
> multiple asynchronous calls to produce.  You could have callbacks that don't do
> anything if there is nothing to consume (sort of null objects I guess).

Twisted is interesting and very powerful but requires a different way
of thinking about the problem and designing a solution. More to the
point, callbacks often provide a less flexible and simple API than an
iterator that yields results (consumed items). For example, say that
you want to store the results to a dictionary. Using callbacks, you
would have to explicitly synchronize each access to the dictionary
since they may fire independently. OTOH an iterator by definition
yields items sequentially, so the client doesn't have to bother with
synchronization. Note that with "client" I mean the user of an API/
framework/library; the implementation of the library itself may of
course use callbacks under the hood (e.g. to put incoming results to a
Queue) but expose the API as a simple iterator.

George