thread help

[Roger Binns]

Scott David Daniels wrote:
> An alternative is to create a que into which you push IP addresses to
> contact, and have each thread read addresses off the queue when they are
> free to process them.  This has the advantage of decoupling the number
> of threads from the number of addresses you want to examine.

That is also the general best design pattern for doing threading.
Have a Queue.Queue object where you place work items and have
threads pull items off the queue and execute it.  You can use
callbacks or another Queue for placing the results on.

The Queue.Queue class has the nice property that it is very
thread safe, and you can do both blocking and timed waits
on it.

The other problem to deal with that is particular to Python is
how to stop threads when you want to shutdown or cancel actions.
At the very least you can have a 'shutdown' message you place
on the Queue that when any thread reads, it shuts down.

Unfortunately Python doesn't allow interrupting a thread,
so any thread doing something will run to completion.  You
can check a variable or something in lines of Python
code, but cannot do anything when in C code.  For example
if you do some networking stuff and the C code (eg a DNS
lookup followed by a TCP connect) takes 2 minutes, then
you will have to wait at least that long.

In the simplest case you can just make all your threads be
daemon.  Python will shutdown when there are no non-daemon
threads left, so you can just exit your main loop and all
will shutdown.  However that means the worker threads just
get abruptly stopped in the middle of what they were
doing.

(IMHO it would be *really* nice if Python provided a way
to interrupt threads).

Roger