Producer-consumer threading problem

[George Sakkis]

On Jun 11, 10:13 am, Jean-Paul Calderone <exar... at divmod.com> wrote:
> On Tue, 10 Jun 2008 22:46:37 -0700 (PDT), George Sakkis <george.sak... at gmail.com> wrote:
> >On Jun 10, 11:47 pm, Larry Bates <larry.ba... at websafe.com`> wrote:
>
> >> I had a little trouble understanding what exact problem it is that you are
> >> trying to solve but I'm pretty sure that you can do it with one of two methods:
>
> >Ok, let me try again with a different example: I want to do what can
> >be easily done with 2.5 Queues using Queue.task_done()/Queue.join()
> >(see example athttp://docs.python.org/lib/QueueObjects.html), but
> >instead of  having to first put all items and then wait until all are
> >done, get each item as soon as it is done.
>
> >> 1) Write the producer as a generator using yield method that yields a result
> >> every time it is called (something like os.walk does).  I guess you could yield
> >> None if there wasn't anything to consume to prevent blocking.
>
> >Actually the way items are generated is not part of the problem; it
> >can be abstracted away as an arbitrary iterable input. As with all
> >iterables, "there are no more items" is communicated simply by a
> >StopIteration.
>
> >> 2) Usw somethink like Twisted insted that uses callbacks instead to handle
> >> multiple asynchronous calls to produce.  You could have callbacks that don't do
> >> anything if there is nothing to consume (sort of null objects I guess).
>
> >Twisted is interesting and very powerful but requires a different way
> >of thinking about the problem and designing a solution. More to the
> >point, callbacks often provide a less flexible and simple API than an
> >iterator that yields results (consumed items). For example, say that
> >you want to store the results to a dictionary. Using callbacks, you
> >would have to explicitly synchronize each access to the dictionary
> >since they may fire independently.
>
> This isn't true.  Access is synchronized automatically by virtue of the
> fact that there is no pre-emptive multithreading in operation.  This is
> one of the many advantages of avoiding threads. :)  There may be valid
> arguments against callbacks, but this isn't one of them.
>
> Jean-Paul

Thanks for the correction; that's an important advantage for callbacks
in this case.

George