Improving the web page download code.

[mukesh tiwari]

On Wednesday, 28 August 2013 04:03:15 UTC+5:30, MRAB  wrote:
> On 27/08/2013 21:53, mukesh tiwari wrote:
> 
> > On Wednesday, 28 August 2013 01:49:59 UTC+5:30, MRAB  wrote:
> 
> >> On 27/08/2013 20:41, mukesh tiwari wrote:
> 
> >>
> 
> [snip]
> 
>  >> > if __name__== '__main__':
> 
>  >> > 	u = Downloader()
> 
>  >> > 	signal.signal( signal.SIGINT , u.handleexception)
> 
>  >> > 	thread.start_new_thread ( u.createurl , () )
> 
>  >> > 	for i in xrange ( 5 ) :
> 
>  >> > 		thread.start_new_thread ( u.downloadurl , () )
> 
>  >> > 	while True : pass
> 
>  >> > 			
> 
>  >> >
> 
>  >> My preferred method when working with background threads is to put a
> 
>  >> sentinel such as None at the end and then when a worker gets an item
> 
>  >> from the queue and sees that it's the sentinel, it puts it back in
> 
>  >> the queue for the other workers to see, and then returns
> 
>  >> (terminates). The main thread can then call each worker thread's
> 
>  >> .join method to wait for it to finish. You currently have the main
> 
>  >> thread running in a 'busy loop', consuming processing time doing
> 
>  >> nothing!
> 
>  >
> 
>  > Hi MRAB,
> 
>  > Thank you for the reply. I wrote this while loop only because of
> 
>  > there is no thread.join in thread[1] library but I got your point. I
> 
>  > am simply running a while loop for doing nothing. So if somehow I can
> 
>  > block the main without too much computation then it will great.
> 
>  >
> 
> Why don't you use the 'threading' module instead?
> 
> 
> 
> 
> 
> creator = threading.Thread(target=u.createurl)
> 
> 
> 
> workers = []
> 
> for i in xrange(5):
> 
> 	workers.append(threading.Thread(target=u.downloadurl))
> 
> 
> 
> creator.start()
> 
> 
> 
> for w in workers:
> 
> 	w.start()
> 
> 
> 
> creator.join()
> 
> 
> 
> for w in workers:
> 
> 	w.join()

Hi MRAB,
Initially I blocked the main using raw_input('') and it was working fine. 

u = Downloader()
signal.signal( signal.SIGINT , u.handleexception)
thread.start_new_thread ( u.createurl , () )
for i in xrange ( 5 ) :
            thread.start_new_thread ( u.downloadurl , () )
#This is for blocking main
raw_input('')
When I pressed  ctrl-c then it's responding fine but now after switching to threading module, I am not able to kill my program using SIGINT ( ctrl-c ). Any idea how to signal SIGINT to threads ? 

Now the changed code and I have to catch the SIGINT. 
        u = Downloader()
        signal.signal( signal.SIGINT , u.handleexception)
        urlcreator = threading.Thread ( target = u.createurl )
        
        workers = []
        for i in xrange ( 5 ):
                workers.append ( threading.Thread( target = u.downloadurl ) )

        urlcreator.start()
        for w in workers:
                w.start()
        
        urlcreator.join()
        for w in workers:
                w.join()

-Mukesh Tiwari