[Catalog-sig] Cheese Shop Meltdown

[Rob Cakebread]

Until the problems with "dumb spiders"[1] are worked around, couldn't
something simple
be done such as using a spider trap[2]?

Add this to robots.txt:

User-agent: *
Disallow: /pypi?:action=bot_ban

Add a hidden link on the front page, then grab the ip of the offending
spider and block it by adding
it to .htacess

Just a thought, as PyPI has been painfully slow for hours at a time lately.


[1] http://mail.python.org/pipermail/catalog-sig/2007-April/001055.html
[2] http://danielwebb.us/software/bot-trap/