[New-bugs-announce] [issue13281] robotparser.RobotFileParser ignores rules preceeded by a blank line

[Brian Bernstein]

New submission from Brian Bernstein <bernie9998 at gmail.com>:

When attempting to parse a robots.txt file which has a blank line between allow/disallow rules, all rules after the blank line are ignored.

If a blank line occurs between the user-agent and its rules, all of the rules for that user-agent are ignored.

I am not sure if having a blank line between rules is allowed in the spec, but I am seeing this behavior in a number of sites, for instance:

http://www.whitehouse.gov/robots.txt has a blank line between the disallow rules all other lines, including the associated user-agent line, resulting in the python RobotFileParser to ignore all rules.

http://www.last.fm/robots.txt appears to separate their rules with arbitrary blank lines between them.  The python RobotFileParser only sees the first two rule between the user-agent and the next newline.

If the parser is changed to simply ignore all blank lines, would it have any adverse affect on parsing robots.txt files?

I am including a simple patch which ignores all blank lines and appears to find all rules from these robots.txt files.

----------
files: robotparser.py.patch
keywords: patch
messages: 146518
nosy: bernie9998
priority: normal
severity: normal
status: open
title: robotparser.RobotFileParser ignores rules preceeded by a blank line
type: behavior
versions: Python 2.7
Added file: http://bugs.python.org/file23538/robotparser.py.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13281>
_______________________________________