Thread Question

[Justin Azoff]

Ritesh Raj Sarraf wrote:
> I'd like to put my understanding over here and would be happy if people can
> correct me at places.
ok :-)

> So here it goes:
> Firstly the code initializes the number of threads. Then it moves on to
> initializing requestQueue() and responseQueue().
> Then it moves on to thread_pool, where it realizes that it has to execute the
> function run().
> >From NUMTHREADS in the for loop, it knows how many threads it is supposed to
> execute parallelly.

right...

> So once the thread_pool is populated, it starts the threads.
> Actually, it doesn't start the threads. Instead, it puts the threads into the
> queue.

Neither.. it puts the threads into a list.  It "puts" the queues into
the thread - by passing them as arguments to the Thread constructor.

> Then the real iteration, about which I was talking in my earlier post, is done.
> The iteration happens in one go. And requestQueue.put(item) puts all the items
> from lRawData into the queue of the run().

It doesn't necessarily have put all the items into the queue at once.
The previous line starts all the threads, which immediately start
running
    while 1:
        item = request.get()
        ....

the default Queue size is infinite, but the program would still work
fine if the queue was fixed to say, 6 elements.  Now that I think of
it, it may even perform better... If you have an iterator that will
generate a very large number of items, and the function being called by
each thread is slow, the queue may end up growing to hold millions of
items and cause the system to run out of memory.

> But there, the run() already known its limitation on the number of threads.

run() doesn't know anything about threads.  All it knows is that it can
call request.get() to get an item to work on, and response.put() when
finished.

> No, I think the above statement is wrong. The actual pool about the number of
> threads is stored by thread_pool. Once its pool (at a time 3 as per this
> example) is empty, it again requests for more threads using the requestQueue()

The pool is never empty. The program works like a bank with 3 tellers.
Each teller knows nothing about any of the other tellers, or how many
people are waiting in the line.  All they know is that when they say
"Next!" (request.get()) another person steps in front of them.  The
tellers don't move, the line moves.

> And in function run(), when the item of lRawData is None, the thread stops.
> The the cleanup and checks of any remaining threads is done.

Yes, since each thread doesn't know anything about the rest of the
program, when you send it an empty item it knows to quit.  It would be
analogous to the bank teller saying "Next!" and instead of a customer,
the bank mananger steps forward to tell them that they can go home for
the day.

> Is this all correct ?

Mostly :)  When you understand it fully, you should look at the example
I showed you before.  It is essentially the same thing, just wrapped in
a class to be reusable.

-- 
- Justin