TaskQueue

[Carl Banks]

Raymond Hettinger wrote:
> I would like to get feedback on an idea I had for simplifying the use
> of queues with daemon consumer threads
>
> Sometimes, I launch one or more consumer threads that wait for a task
> to enter a queue and then work on the task. A recurring problem is that
> I sometimes need to know if all of the tasks have been completed so I
> can exit or do something with the result.
>
> If each thread only does a single task, I can use t.join() to wait
> until the task is done.  However, if the thread stays alive and waits
> for more Queue entries, then there doesn't seem to be a good way to
> tell when all the processing is done.

Hmm.  How about this: the Producer can cause an exception to be raised
in any Consumers waiting on the Queue, or vice-versa.

I remember encountering a similar problem when writing a slide viewer
that prefetched and slides.  I wanted to have the ability to terminate
the slide show early.  But what you have is five slides sitting in the
Queue when the user quits (the UI was in the consumer thread, of
course).  Now how do you signal the Producer thread to exit?  It seemed
to me that allowing one thread to raise an exception in another was the
most straightforward way to do this.  (In my case is was the Consumer
raising it in the Producer.)

I briefly considered a subclass of Queue to implement it, but it was
too difficult a thing that I would have wanted to work on at that point
in time.  I implemented some workaround that I don't remember, and
forgot about the issue until your post.

But yeah, something like an InterruptableQueue might be a nice thing to
have.


Carl Banks