How to use threading and Queue modules

["Martin v. Löwis"]

Paul Moore wrote:
> But now I hit a problem: How do I wait for the workers to all finish?
> If I just join all the workers in turn, I risk the queue filling up,
> resulting in deadlock. If I wait on the queue, I don't know when all
> the workers have finished.

Is it the case that you have one result per worker? The I recommend that 
each worker puts a pair (thread, thread-result) into the queue. The main 
thread waits on the queue, picks up the result, and joins the worker thread.

If there are multiple results per worker, add a flag indicating that the 
result you got is the last result.

> Maybe a queue isn't the right data structure here. Is there another,
> more suitable, data structure which can collect results from worker
> threads? Maybe all I want is a locked list. After all, the queue would
> be fine if it couldn't fill up. (There isn't enough data here for it
> to be a problem to keep it all in memory).

Ah, so you are concerned that the queue may fill up? Don't be: If you 
don't give a maximum size for queue construction, the queue is unbounded.

Regards,
Martin