[issue13924] Mercurial robots.txt should let robots crawl landing pages.

[Antoine Pitrou]

Antoine Pitrou added the comment:

Yes, I think we should whitelist rather than blacklist. The problem with letting engines index the repositories is the sheer resource cost when they fetch many heavy pages (such as annotate, etc.).

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13924>
_______________________________________