[New-bugs-announce] [issue28756] robotfileparser always uses default Python user-agent

[John Nagle]

New submission from John Nagle:

urllib.robotparser.RobotFileParser always uses the default Python user agent. This agent is now blacklisted by many sites, and it's not possible to read the robots.txt file at all.

----------
components: Library (Lib)
messages: 281314
nosy: nagle
priority: normal
severity: normal
status: open
title: robotfileparser always uses default Python user-agent
type: enhancement
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28756>
_______________________________________