Queue cleanup

[Paul Rubin]

EW <ericwoodworth at gmail.com> writes:
> I also might have different consumer threads do
> different tasks (for example one might write to a log and one might
> write to SQL) so that again means I can't plan for a set ratio of
> consumers to producers....  So it's unknown.
>
> So this means that instead of having 1 Queue that all the producers
> put to and that all the consumers get from I actually have 1 Queue per
> producer thread 

That doesn't sound appropriate.  Queues can have many readers and many
writers.  So use one queue per task (logging, SQL, etc), regardless of
the number of producer or consumer threads.  Any producer with an SQL
request sends it to the SQL queue, which can have many listeners.  The
different SQL consumer threads listen to the SQL queue and pick up
requests and handle them.