Thread processing

[Dave Brueck]

On Wed, 5 Mar 2003, Paul Moore wrote:

> I've just started working with threads in Python, and I'm having a bit
> of a hard time getting my head round the concepts.

Hi Paul,

Your question seems very similar to the one you asked yesterday, so even
though you're reluctant to go into too much detail, maybe a specific
example will help you get the answer you're looking for.

> I can use a Queue to have the threads notify when they have finished,
> but that means rewriting the thread procedures to do the notification,
> which isn't really what I want. Something like:
>
>     q = Queue()
>     # All the threads need to get q as input, and do
>     # q.put(self) when the run method finishes
>
>     while threads:
>         thread = q.get()
>         thread.join()

You don't need to join the thread if it is just going to terminate on its
own.

>         threads.remove[thread]

Do you actually need a list of all the threads? Why bother?

>         # post-processing for thread here

This is what you need to clarify - what sort of post-processing needs to
be done? Do you really need to have a handle to the thread object or just
the data to be post-processed?

> That's nasty because I have to write my thread procedures specifically
> to work with this idiom (and I'm trying to make this stuff as generic
> as possible, because I have a lot of potential code which will follow
> this pattern)

Is it really _that_ big of a deal to have the worker thread put done work
on a queue? Short of coming up with a better idiom, having threads place
finished work onto a Queue is very flexible and generic - easily reusable.
Also, the model extends quite well to scenarios where each thread does
more than one piece of work before terminating - a pool of workers that
pop data of one queue, do stuff, and then place the finished work on the
done queue.

> Am I right in thinking that there isn't a good way of doing this? If
> anyone can point out what I might have missed, I'd be very grateful.

A common pattern for a pool of worker threads is to dump finished work
into a queue. IMO that _is_ a good way of doing this as it is very
flexible, not problematic, and very simple to write in Python.

> PS Sorry I can't give a specific example - I'm trying to write a
>    fairly generic library here - I have 4 main use cases already, with
>    more appearing fairly fast... (Actually, I could probably describe
>    my particular current problem if that would help, but this message
>    is already too long, and I don't want to lose the general question
>    in details of the one specific case...)

Maybe the specific case would shed light on what you don't like.

-Dave