[issue17403] Robotparser fails to parse some robots.txt
Mher Movsisyan
report at bugs.python.org
Tue Mar 19 08:11:22 CET 2013
Mher Movsisyan added the comment:
The second patch only normalizes the url. From http://www.robotstxt.org/norobots-rfc.txt it is not clear how to handle multiple rules with the same prefix.
----------
Added file: http://bugs.python.org/file29476/parser2.patch
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17403>
_______________________________________
More information about the Python-bugs-list
mailing list