search an entire website given the homepage URL

[Fredrik Lundh]

"Bell, Kevin" wrote:

> >use a search engine (try the search box in the upper right corner).
>
> >using a spider to download the entire site just so you can "search
> through >it" is bloody impolite.
>
> Really?  I'd argue that's impolite only if you're an impolite person
> with a rude agenda, which is not what I had in mind, but thanks for the
> ethics lecture as well as the pointer ; )

downloading entire sites for personal purposes is considered rude by
many web administrators, no matter what agenda you have.

> I assure you that I harbor no nefarious scheme.  Isn't it common for folks
> to watch the stock market, or real estate listings, for example?

to grab entire sites ?

try doing that on a commercial data provider's site, and chances are
that you'll end up being banned (or sued) within hours ...

</F>