[issue44209] urllib.robotparser fail on Disallow: /? from google.com

[Karl Y. Pradene]

New submission from Karl Y. Pradene <karl.pradene at gmail.com>:

In robotparser.py
On line 222
path = urllib.parse.urlunparse(urllib.parse.urlparse(path))
tranform the entry Disallow: /?
in the google.com/robots.txt
in : Disallow: /
making every can_fetch request return False

----------
components: Library (Lib)
messages: 394144
nosy: karl.pradene
priority: normal
severity: normal
status: open
title: urllib.robotparser fail on Disallow: /? from google.com
type: behavior
versions: Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue44209>
_______________________________________