Thread Question

[Simon Forman]

Ritesh Raj Sarraf wrote:
> Duncan,
>
> I couldn't make out much from the code.

Please, try again to understand Duncan's code.  It's much better than
what you did.

> Instead this is what I did.
>
> threads = []
>         nloops = range(len(lRawData))
>         for i in nloops:
>             (sUrl, sFile, download_size, checksum) =
> stripper(lRawData[i])
>             t = threading.Thread(target=download_from_web, args=(sUrl,
> sFile, sSourceDir, None))
>             # = pypt_thread(download_from_web, i,
> stripper(lRawData[i]))
>             threads.append(t)
>
>         i = 0
>         join_i = 0
>         while i < nloops:
>             counter = 0
>             while counter < 3:
>                 threads[i].start()
>                 counter += 1
>                 i += 1
>             counter = 0
>             join_i = i - 3
>             while counter < 3:
>                 threads[join_i].join()
>                 counter += 1
>                 join_i += 1
>
> Is this correct ? Comments!!

This is just painful.  It's not exactly "incorrect", but I think (I
*think*, it's hard to tell even after reading it 3 times) that it's
going to only download three requests at a time and if one (or more) of
the requests takes a long time it will hold up all the others.  Also,
you're creating one thread per item in lRawData even though you only
need/use three at a time.

if you set numthreads = 3 in Duncan's code, you would only be
downloading 3 things at a time, but you'd only be using three threads
and long running requests would not affect the others.

If you need help understanding it please ask questions.  I, for one,
would be happy to comment it for you to explain how it works.  It's so
nice and elegant that I've already cut-and-pasted it into my own
"notebook" of cool useful python "patterns" to use in the future.

Reread it slowly, think about what it's doing, if questions arise write
them down and ask them.  Duncan's code is beautiful.  It's well worth
your time to understand it.

Peace,
~Simon

>
> Ritesh
>
>
> Duncan Booth wrote:
> > Ritesh Raj Sarraf wrote:
> >
> > > I'm planning to implement threads in my application so that multiple
> > > items can be downloaded concurrently. I want the thread option to be
> > > user-defined.
> > >
> > > Looking at the documentation of threads (Core Python Programming), I've
> > > noticed that all threads are executed a once. Depending upon what they
> > > are doing, some finish early and some later.
> > >
> > > But I want to implement something like:
> > >
> > > for item in list_items:
> > >     for num in thread_args:
> > >        thread[num].start()
> > >        thread[num].start()
> > >
> > > Is this the correct way of threading applications ?
> > > This is the first time I'd be doing threading. So was looking for
> > > comments and suggestions.
> > >
> >
> > What you want is to use a pool of threads so that you can configure how
> > many requests are issued at a time (you don't want to try to issue 100
> > requests all in parallel). You can communicate with the threads through a
> > Queue.
> >
> > So if the code for a thread looks like:
> >
> >    def run(request, response):
> >        while 1:
> >            item = request.get()
> >            if item is None:
> >                 break
> >            response.put(download_from_web(item))
> >
> > # your main loop can be something like:
> >
> > requestQueue = Queue()
> > responseQueue = Queue()
> > thread_pool = [
> >         Thread(target=run, args=(requestQueue, responseQueue)
> >         for i in range(numthreads)]
> > for t in thread_pool: t.start()
> >
> > for item in list_items:
> >      requestQueue.put(item)
> >
> > for i in range(len(list_items)):
> >     response = responseQueue.get()
> >     handle_response(response)
> >
> > # and then to shut down the threads when you've finished:
> > for t in thread_pool:
> >     requestQueue.put(None)
> > for t in thread_pool:
> >      t.join()