[Module] RobotRules

[Vivake Gupta]

                               RobotRules                               
                               ----------                               

Parses robots.txt files.

NB: This is a straight conversion of the perl module WWW::RobotRules

This module parses a robots.txt file as specified in
http://info.webcrawler.com/mak/projects/robots/norobots.html



       URL:  http://nano.com/~vivake/python/
  Download:  http://nano.com/~vivake/python/RobotRules.py

   License:  Python Style

  Categories:  Web

Vivake Gupta (vivake@nano.com)
http://nano.com/~vivake/

--
<a href="http://nano.com/~vivake/python/">RobotRules</a> -- Parses
robots.txt files.