Queue.get_nowait() raises Empty when it's not...

[Fredrik Lundh]

Stephen J. Turner <sjturner at ix.netcom.com> wrote:
> Of course, the documentation for the Queue module says as much, stating
> that Empty is "raised when non-blocking get() (or get_nowait()) is
> called on a Queue object which is empty *or locked* [emphasis mine]."
> 
> If I understand the implementation correctly, this indeterminate
> condition is because, in order to see if the queue is empty,
> get_nowait() tries to acquire (without waiting) the "empty" semaphore. 
> If the lock acquisition fails, it means that either:
> 
> - The queue is truly empty, since the last successful caller of
> get_nowait() to remove an item from the queue would have returned
> without releasing the "empty" semaphore.
> 
> - Another thread is also in the get_nowait() method and currently holds
> the "empty" semaphore.
> 
> This behavior poses a problem for an application of mine.  I have a set
> of tasks to be executed by a pool of worker threads.  The number of
> tasks is known at the time the application is run.  The main thread
> reads in each task from file, checks it for errors, then puts it on a
> work queue.  Once all tasks are validated and enqueued, the main thread
> then creates all of the worker threads (using the threading module) and
> exits.  Each worker thread continues to call get_nowait() until it sees
> the Empty exception, then exits.  Once all worker threads exit, the
> process itself exits.
> 
> The problem, then, is that if two or more worker threads call
> get_nowait() simultaneously, all but one will be thrown an Empty
> exception and so assume (erroneously) that they can leave.

easy way out: add a sentinel (typically None) to the queue
for each worker thread, and have the worker threads to
terminate when they read the sentinel value from the queue.

</F>

<!-- (the eff-bot guide to) the standard python library:
http://www.pythonware.com/people/fredrik/librarybook.htm
-->