web searching scripts

[Avell Diroll]

julien.lord at gmail.com wrote:
> Does anyone know of a freely available script that can take a given URL
> and follow every link within it?
> 
> Ideally, I would like to start with this to build a quick application
> to grab all the content off a website to publish it to a CD.
> 
> Thanks,
> 
> jul
> 


If you just want to download websites (i.e. not necessarily writing a
program yourself to do that), you may try Httrack, it might suite your
needs.

http://www.httrack.com/

There even seem to be some sort of python bindings ...

http://www.satzbau-gmbh.de/staff/abel/httrack-py/

But there might be some more pythonic solution around ... i would start
looking at twisted or cherrypy, but i never used them myself ...

HIH

regards

Avell