Thread processing

[Paul Moore]

I've just started working with threads in Python, and I'm having a bit
of a hard time getting my head round the concepts.

One thing I can't work out how to do is to wait on a group of threads
and handle each one as it finishes. In Windows, I'd use
WaitForMultipleObjects(), but I can't find an equivalent for Python
threads.

I can join the threads in an arbitrary order

    for thread in threads:
        thread.join()
        # post-processing for thread here

but this doesn't let me handle each thread as it finishes. This can be
a problem, as one of my use cases is multiplexing 50 or more database
connections. Some of the connecions may fail with a timeout, taking
quite a long time to return an error. It would be a shame if the fast,
working connections had to wait behind one slow failure.

I can busy-wait, something like

    running = True
    while running:
        for thread in threads:
            if not thread.isAlive():
                thread.join()
                # post-processing for thread here

but I'm pretty sure busy waiting isn't the best way of dealing with
this...

I can use a Queue to have the threads notify when they have finished,
but that means rewriting the thread procedures to do the notification,
which isn't really what I want. Something like:

    q = Queue()
    # All the threads need to get q as input, and do
    # q.put(self) when the run method finishes

    while threads:
        thread = q.get()
        thread.join()
        threads.remove[thread]
        # post-processing for thread here

That's nasty because I have to write my thread procedures specifically
to work with this idiom (and I'm trying to make this stuff as generic
as possible, because I have a lot of potential code which will follow
this pattern)

I can't even think of a way of subclassing Thread, as the Thread class
is designed for (user) subclasses to override run(), so there's
nowhere I can put a "the user code has finished" hook, short of
digging into the internals.

Am I right in thinking that there isn't a good way of doing this? If
anyone can point out what I might have missed, I'd be very grateful.

In the absence of a simple WaitForMultipleObjects equivalent, I'll
probably go with a mixture of the join-in-arbitrary-order and the
Queue approaches. But neither really works for all the cases I care
about...

Thanks,
Paul.

PS Sorry I can't give a specific example - I'm trying to write a
   fairly generic library here - I have 4 main use cases already, with
   more appearing fairly fast... (Actually, I could probably describe
   my particular current problem if that would help, but this message
   is already too long, and I don't want to lose the general question
   in details of the one specific case...)
-- 
This signature intentionally left blank