[issue17403] Robotparser fails to parse some robots.txt

Mher Movsisyan report at bugs.python.org
Tue Mar 19 08:11:22 CET 2013


Mher Movsisyan added the comment:

The second patch only normalizes the url. From http://www.robotstxt.org/norobots-rfc.txt it is not clear how to handle multiple rules with the same prefix.

----------
Added file: http://bugs.python.org/file29476/parser2.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17403>
_______________________________________


More information about the Python-bugs-list mailing list