Thread Question

[Duncan Booth]

Ritesh Raj Sarraf wrote:

> I'm planning to implement threads in my application so that multiple
> items can be downloaded concurrently. I want the thread option to be
> user-defined.
> 
> Looking at the documentation of threads (Core Python Programming), I've
> noticed that all threads are executed a once. Depending upon what they
> are doing, some finish early and some later.
> 
> But I want to implement something like:
> 
> for item in list_items:
>     for num in thread_args:
>        thread[num].start()
>        thread[num].start()
> 
> Is this the correct way of threading applications ?
> This is the first time I'd be doing threading. So was looking for
> comments and suggestions.
> 

What you want is to use a pool of threads so that you can configure how 
many requests are issued at a time (you don't want to try to issue 100 
requests all in parallel). You can communicate with the threads through a 
Queue.

So if the code for a thread looks like:

   def run(request, response):
       while 1:
           item = request.get()
           if item is None:
                break
           response.put(download_from_web(item))

# your main loop can be something like:

requestQueue = Queue()
responseQueue = Queue()
thread_pool = [
        Thread(target=run, args=(requestQueue, responseQueue)
        for i in range(numthreads)]
for t in thread_pool: t.start()

for item in list_items:
     requestQueue.put(item)

for i in range(len(list_items)):
    response = responseQueue.get()
    handle_response(response)

# and then to shut down the threads when you've finished:
for t in thread_pool:
    requestQueue.put(None)
for t in thread_pool:
     t.join()