How to use threading and Queue modules

[Paul Moore]

I have a problem trying to use the threading and Queue modules. What
I'm trying to do is to fire off a host of background threads, and let
them do some work. In my main thread, I want to wait for the
background workers to complete, and then pick up their results.

The best way to handle the results seems to be via a Queue.Queue, so
that I don't have to worry about locking issues.

But now I hit a problem: How do I wait for the workers to all finish?
If I just join all the workers in turn, I risk the queue filling up,
resulting in deadlock. If I wait on the queue, I don't know when all
the workers have finished.

Maybe a queue isn't the right data structure here. Is there another,
more suitable, data structure which can collect results from worker
threads? Maybe all I want is a locked list. After all, the queue would
be fine if it couldn't fill up. (There isn't enough data here for it
to be a problem to keep it all in memory).

I'm new to multi-threading, so I don't want something too hard :-)

Basically, the only reason I'm using threads at all is to multiplex a
lot of database conversations (50 or more) where IO and network times
are the bottleneck - with threading, I can run all 50 in the time it
takes the slowest to connect!

Thanks for any suggestions.

Paul.