Producer-consumer threading problem

[Jean-Paul Calderone]

On Tue, 10 Jun 2008 22:46:37 -0700 (PDT), George Sakkis <george.sakkis at gmail.com> wrote:
>On Jun 10, 11:47 pm, Larry Bates <larry.ba... at websafe.com`> wrote:
>>
>> I had a little trouble understanding what exact problem it is that you are
>> trying to solve but I'm pretty sure that you can do it with one of two methods:
>
>Ok, let me try again with a different example: I want to do what can
>be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
>(see example at http://docs.python.org/lib/QueueObjects.html), but
>instead of  having to first put all items and then wait until all are
>done, get each item as soon as it is done.
>
>> 1) Write the producer as a generator using yield method that yields a result
>> every time it is called (something like os.walk does).  I guess you could yield
>> None if there wasn't anything to consume to prevent blocking.
>
>Actually the way items are generated is not part of the problem; it
>can be abstracted away as an arbitrary iterable input. As with all
>iterables, "there are no more items" is communicated simply by a
>StopIteration.
>
>> 2) Usw somethink like Twisted insted that uses callbacks instead to handle
>> multiple asynchronous calls to produce.  You could have callbacks that don't do
>> anything if there is nothing to consume (sort of null objects I guess).
>
>Twisted is interesting and very powerful but requires a different way
>of thinking about the problem and designing a solution. More to the
>point, callbacks often provide a less flexible and simple API than an
>iterator that yields results (consumed items). For example, say that
>you want to store the results to a dictionary. Using callbacks, you
>would have to explicitly synchronize each access to the dictionary
>since they may fire independently.

This isn't true.  Access is synchronized automatically by virtue of the
fact that there is no pre-emptive multithreading in operation.  This is
one of the many advantages of avoiding threads. :)  There may be valid
arguments against callbacks, but this isn't one of them.

Jean-Paul