[issue21475] Support the Sitemap extension in robotparser

[Berker Peksag]

Berker Peksag added the comment:

The Crawl-delay part(issue 16099) is now committed.

----------
stage:  -> needs patch
title: Support the Sitemap and Crawl-delay extensions in robotparser -> Support the Sitemap extension in robotparser
versions: +Python 3.6 -Python 3.5

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21475>
_______________________________________