[issue17403] Robotparser fails to parse some robots.txt

[R. David Murray]

R. David Murray added the comment:

Lucaz pointed out on IRC that the problem is that the current robotparser is implementing an outdated robots.txt standard.  He may work on fixing that.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17403>
_______________________________________