Problem with Python's "robots.txt" file parser in module robotparser

[Nikita the Spider]

In article <yPCli.46145$5j1.13795 at newssvr21.news.prodigy.net>,
 John Nagle <nagle at animats.com> wrote:

>    I asked over at Webmaster World, and over there, they recommend against
> using redirects on robots.txt files, because they questioned whether all of
> the major search engines understand that.  Does a redirect for 
> "foo.com/robots.txt" mean that the robots.txt file applies to the domain
> being redirected from, or the domain being redirected to?

Good question. I'd guess the latter, but it's a little ambiguous. I 
agree that redirecting a request for robots.txt is probably not a good 
idea. Given that the robots.txt standard isn't as standard as it could 
be, I think it's a good idea in general to apply the KISS principle when 
dealing with things robots.txt-y.

-- 
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more