Writing a Web Robot in Python

[mudgen at gmail.com]

I'm working on writing a web robot (for searching web pages) in Python
so I've been browsing around the web looking for data about building
web robots.

I'm surprised to find that I didn't find much data. Seems like robots
would be kind of a common thing to build. I know there is this: "The
Web Robots Pages" (http://www.robotstxt.org/wc/robots.html) but that
doesn't have a lot of data and seems pretty out of date.

I checked amazon.com for books on building web robots and I didn't find
very much. Seems kind of weird that I don't find much data on this
subject. Anybody know some good places to go to get information on
building a web robots?