Producer-consumer threading problem

[Rhamphoryncus]

On Jun 11, 6:00 am, George Sakkis <george.sak... at gmail.com> wrote:
> On Jun 11, 1:59 am, Rhamphoryncus <rha... at gmail.com> wrote:
>
> > Why not use a normal Queue, put a dummy value (such as None) in when
> > you're producer has finished, and have the main thread use the normal
> > Thread.join() method on all your child threads?
>
> I just gave two reasons:
> - Concurrency / interactivity. The main thread shouldn't wait for all
> one million items to be produced to get to see even one of them.

Then don't wait.  The main thread can easily do other work while the
producer and consumer threads go about their business.


> - Limiting resources. Just like iterating over the lines of a file is
> more memory efficient than reading the whole file in memory, getting
> each consumed item as it becomes available is more memory efficient
> than waiting for all of them to finish.

That's why you give Queue a maxsize.  Put it at maybe 5 or 10.  Enough
that the producer can operate in a burst (usually more efficient that
switching threads after each item), then the consumer can grab them
all in a burst.

Then again, you may find it easier to use an event-driven architecture
(like Twisted, as others have suggested.)