Queue cleanup

[EW]

On Aug 11, 12:55 pm, EW <ericwoodwo... at gmail.com> wrote:
> Hi
>
> I'm writing a multithreaded app that relies on Queues to move data
> between the threads.  I'm trying to write my objects in a general way
> so that I can reuse them in the future so I need to write them in such
> a way that I don't know how many producer and how many consumer
> threads I might need.  I also might have different consumer threads do
> different tasks (for example one might write to a log and one might
> write to SQL) so that again means I can't plan for a set ratio of
> consumers to producers.  So it's unknown.
>
> So this means that instead of having 1 Queue that all the producers
> put to and that all the consumers get from I actually have 1 Queue per
> producer thread  that the main body sends to the correct type of
> consumer thread.  So I could get something like this where 3 producer
> threads write to 3 different Queues all of which get read by 1
> consumer thread:
>
> P1    P2   P3
>      \    |   /
>        \  |  /
>         C1
>
> So producers 1, 2, and 3 all write to individual Queues and consumer 1
> had a list of those Queues and reads them all.  The problem I'm having
> is that those producer threads can come and go pretty quickly and when
> they die I can cleanup the thread with join() but I'm still left with
> the Queue.  So I could get something like this:
>
> P1         P3
>      \    |   /
>        \  |  /
>         C1
>
> So here the P2 thread has ended and gone away but I still have his
> Queue lingering.
>
> So on a thread I can use is_alive() to check status and use join() to
> clean up but I don't see any analogous functionality for Queues.  How
> do I kill them?  I thought about putting a suicide message on the
> Queue and then C1 would read it and set the variable to None but i'm
> not sure setting the variable to None actually makes the Queue go
> away.  It could just end up sitting in memory unreferenced - and
> that's not good.  Additionally, I could have any number of consumer
> threads reading that Queue so once the first one get the suicide note
> the other consumer threads never would.
>
> I figure there has to be an elegant way for managing my Queues but so
> far I can't find it.  Any suggestions would be appreciated and thanks
> in advance for any help.
>
> ps Python rocks.

Whoo..the formatting got torn up!  My terrible diagrams are even more
terrible!  Oh well, I think you'll catch my meaning   :)