[issue36207] robotsparser deny all with some rules
wats0ns
report at bugs.python.org
Sat Dec 11 19:08:06 EST 2021
wats0ns <quentin.maire at protonmail.ch> added the comment:
I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented now in most parsers ?
----------
nosy: +quentin-maire
_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36207>
_______________________________________
More information about the Python-bugs-list
mailing list