[issue36207] robotsparser deny all with some rules

[adiboo adib]

adiboo adib <adiboo67 at gmail.com> added the comment:

I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: https://www.st-info.fr/robots.txt, I believe that this is how it's implemented now in most parsers ?

----------
nosy: +adiboo67

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue36207>
_______________________________________