[issue38067] Add headers parameter on RobotFileParser

[Emmanuel Arias]

New submission from Emmanuel Arias <emmanuelarias30 at gmail.com>:

Hello I open an issue from a PR that I think need discussion:

I write here the @fuwaraido's PR description:

> I found some website like "sony.co.jp" requests User-Agent field in headers when I try to fetch their robots.txt. Unless they returns 403 Forbidden status code and RobotTextParser fails to read robots.txt. Currently there is no way to specify headers so I propose add headers arguments in constructor.

----------
components: Library (Lib)
messages: 351440
nosy: eamanu
priority: normal
pull_requests: 15423
severity: normal
status: open
title: Add headers parameter on RobotFileParser
type: enhancement
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue38067>
_______________________________________