Improving the web page download code.

[MRAB]

On 27/08/2013 21:53, mukesh tiwari wrote:
> On Wednesday, 28 August 2013 01:49:59 UTC+5:30, MRAB  wrote:
>> On 27/08/2013 20:41, mukesh tiwari wrote:
>>
[snip]
 >> > if __name__== '__main__':
 >> > 	u = Downloader()
 >> > 	signal.signal( signal.SIGINT , u.handleexception)
 >> > 	thread.start_new_thread ( u.createurl , () )
 >> > 	for i in xrange ( 5 ) :
 >> > 		thread.start_new_thread ( u.downloadurl , () )
 >> > 	while True : pass
 >> > 			
 >> >
 >> My preferred method when working with background threads is to put a
 >> sentinel such as None at the end and then when a worker gets an item
 >> from the queue and sees that it's the sentinel, it puts it back in
 >> the queue for the other workers to see, and then returns
 >> (terminates). The main thread can then call each worker thread's
 >> .join method to wait for it to finish. You currently have the main
 >> thread running in a 'busy loop', consuming processing time doing
 >> nothing!
 >
 > Hi MRAB,
 > Thank you for the reply. I wrote this while loop only because of
 > there is no thread.join in thread[1] library but I got your point. I
 > am simply running a while loop for doing nothing. So if somehow I can
 > block the main without too much computation then it will great.
 >
Why don't you use the 'threading' module instead?


creator = threading.Thread(target=u.createurl)

workers = []
for i in xrange(5):
	workers.append(threading.Thread(target=u.downloadurl))

creator.start()

for w in workers:
	w.start()

creator.join()

for w in workers:
	w.join()