Producer-consumer threading problem

[giltay at gmail.com]

> Sounds like a sentinel would work for this.  The producer puts a
> specific object (say, None) in the queue and the consumer checks for
> this object and stops consuming when it sees it.  But that seems so
> obvious I suspect there's something else up.

     There's a decent implementation of this in the Python Cookbook,
Second Edition (9.4: Working with a Thread Pool), available from
Safari as a preview:
http://my.safaribooksonline.com/0596007973/pythoncook2-CHP-9-SECT-4

     Basically, there's a request_work function that adds (command,
data) pairs to the input Queue.  The command 'stop' is used to
terminate each worker thread (there's the sentinel).
stop_and_free_thread_pool() just puts N ('stop', None) pairs and
join()s each thread.

     The threadpool put()s the consumed items in an output Queue; they
can be retrieved concurrently using get().  You don't call join()
until you want to stop producing; you can get() at any time.

Geoff Gilmour-Taylor

(I ended up using this recipe in my own code, but with a completely
different stopping mechanism---I'm using the worker threads to control
subprocesses; I want to terminate the subprocesses but keep the worker
threads running---and a callback rather than an output queue.)