[issue35457] robotparser reads empty robots.txt file as "all denied"

[Terry J. Reedy]

Terry J. Reedy <tjreedy at udel.edu> added the comment:

https://docs.python.org/2.7/library/robotparser.html#module-robotparser
and
https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
refers users, for file structure, to http://www.robotstxt.org/orig.html.
This says nothing about the effect of an empty file, so I don't see this as a bug.  Even if it was, I would be dubious about reversing the effect without a deprecation notice first, and definitely not in 2.7.

I would propose instead that the doc be changed to refer to the new file, with more and better examples, but add a note that robotparser interprets empty files as 'block all' rather than 'allow all'.

Try bringing this up on python-ideas.

----------
nosy: +terry.reedy
stage:  -> test needed
type: behavior -> enhancement
versions: +Python 3.8 -Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35457>
_______________________________________