[Python-checkins] r62556 - python/trunk/Doc/library/robotparser.rst
skip.montanaro
python-checkins at python.org
Mon Apr 28 05:25:38 CEST 2008
Author: skip.montanaro
Date: Mon Apr 28 05:25:37 2008
New Revision: 62556
Log:
Wrap some long lines.
Modified:
python/trunk/Doc/library/robotparser.rst
Modified: python/trunk/Doc/library/robotparser.rst
==============================================================================
--- python/trunk/Doc/library/robotparser.rst (original)
+++ python/trunk/Doc/library/robotparser.rst Mon Apr 28 05:25:37 2008
@@ -3,7 +3,8 @@
=============================================
.. module:: robotparser
- :synopsis: Loads a robots.txt file and answers questions about fetchability of other URLs.
+ :synopsis: Loads a robots.txt file and answers questions about
+ fetchability of other URLs.
.. sectionauthor:: Skip Montanaro <skip at pobox.com>
@@ -21,8 +22,8 @@
.. class:: RobotFileParser()
- This class provides a set of methods to read, parse and answer questions about a
- single :file:`robots.txt` file.
+ This class provides a set of methods to read, parse and answer questions
+ about a single :file:`robots.txt` file.
.. method:: set_url(url)
@@ -42,20 +43,22 @@
.. method:: can_fetch(useragent, url)
- Returns ``True`` if the *useragent* is allowed to fetch the *url* according to
- the rules contained in the parsed :file:`robots.txt` file.
+ Returns ``True`` if the *useragent* is allowed to fetch the *url*
+ according to the rules contained in the parsed :file:`robots.txt`
+ file.
.. method:: mtime()
- Returns the time the ``robots.txt`` file was last fetched. This is useful for
- long-running web spiders that need to check for new ``robots.txt`` files
- periodically.
+ Returns the time the ``robots.txt`` file was last fetched. This is
+ useful for long-running web spiders that need to check for new
+ ``robots.txt`` files periodically.
.. method:: modified()
- Sets the time the ``robots.txt`` file was last fetched to the current time.
+ Sets the time the ``robots.txt`` file was last fetched to the current
+ time.
The following example demonstrates basic use of the RobotFileParser class. ::
More information about the Python-checkins
mailing list