read all available pages on a Website

Brad Tilley bradtilley at usa.net
Sun Sep 12 20:47:50 EDT 2004


Is there a way to make urllib or urllib2 read all of the pages on a Web 
site? For example, say I wanted to read each page of www.python.org into 
separate strings (a string for each page). The problem is that I don't 
know how many pages are at www.python.org. How can I handle this?

Thanks,

Brad



More information about the Python-list mailing list