Accessing a shared generator from multiple threads.

[Aahz]

In article <400EE6B5.27C6B694 at hotmail.com>,
Alan Kennedy  <alanmk at hotmail.com> wrote:
>
>Yes, you're right. Using a Queue in this situation does require the
>use of a dedicated thread for the producer. There is no way to "pull"
>values from a generator to multiple consumers through a Queue.Queue.
>The values have to be "pushed" onto the Queue.Queue by some producing
>thread of execution.

Correct.

>The way I see it, the options are  
>
>Option 1. Spawn a separate thread to execute the producing generator.
>However, this has problems:-
>
>A: How do the threads recognise the end of the generated sequence?
>This is not a simple problem: the Queue simply being empty does not
>necessarily signify the end of the sequence (e.g., the producer thread
>might not be getting its fair share of CPU time).
>
>B: The Queue acts as a (potentially infinite) buffer for the generated
>values, thus eliminating one of the primary benefits of generators:
>their efficient "generate when required" nature. This can be helped
>somewhat by limiting the number of entries in the Queue, but it is
>still slightly unsatisfactory.
>
>C: A thread of execution has to be dedicated to the producer, thus
>consuming resources.

There are a number of ways of mitigating A and B; they mostly involve
using an extra Queue.Queue to send tokens to the generator thread when a
consumer wants data.  The generator thread then sends back a token that
(among other things) contains an attribute specifically for notifying
the consumer that the generator is exhausted.  See
http://www.pythoncraft.com/OSCON2001/ThreadPoolSpider.py
and
http://www.pythoncraft.com/OSCON2001/FibThreaded.py
for examples that show the technique, though they're not directly
relevant to this case.

My point is that I haven't (yet) seen many good use cases for sharing a
generator between threads, and I'm guessing that many people will try
using generators inappropriately for problems that really are better
suited to Queue.Queue.
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

A: No.
Q: Is top-posting okay?