Thread Question

[Simon Forman]

Ritesh Raj Sarraf wrote:
> Hi,
>
> I have this following situation:
>
>         #INFO: Thread Support
>         # Will require more design thoughts
>         from Queue import Queue
>         from threading import Thread, currentThread
>
>         NUMTHREADS = variables.options.num_of_threads
>
>         def run(request, response, func=download_from_web):
>             '''Get items from the request Queue, process them
>             with func(), put the results along with the
>             Thread's name into the response Queue.
>
>             Stop running once an item is None.'''
>
>             name = currentThread().getName()
>             while 1:
>                 item = request.get()
>                 (sUrl, sFile, download_size, checksum) = stripper(item)
>                 if item is None:
>                     break
>                 response.put((name, func(sUrl, sFile, sSourceDir, None)))
>

One thing about this code: you should check whether item is None
*before* passing it to stripper().  Even if stripper() can handle None
as input there's no reason to make it do so.

> My download_from_web() returns True or False depending upon whether the download
> was successful or failed. How can I check that in the above code ?

Well, you'd probably want to check that *outside* the above run()
function.  The True/False return values will queue up in the
responseQueue, where you can access them with responseQueue.get().
Since these can be in a different order than your requests you're going
to want to "tag" them with the sUrl and sFile.

response.put((name, sUrl, sFile, func(sUrl, sFile, sSourceDir, None)))

That way your response handing code can tell which downloads succeeded
and which failed.

Of course, you could have the run() function check the return value
itself and retry the download ane or a few times.

>
> One other question I had,
> If my user passes the --zip option, download_from_web() internally (when the
> download is successful) zips the downloaded data to a zip file. Since in case
> of threading there'll be multiple threads, and say if one of the thread
> completes 2 seconds before others and is doing the zipping work:
> What will the other thread, at that moment do, if it completes while the
> previous thread is doing the zipping work ?

The other threads will just take the next request from the Queue and
process it.  They won't "care" what the one thread is doing,
downloading, zipping, whatever.


Peace,
~Simon