Web crawler on python

[Support Desk]

-----Original Message-----
From: James Mills [mailto:prologic at shortcircuit.net.au] 
Sent: Sunday, October 26, 2008 5:26 PM
To: sonich
Cc: python-list at python.org
Subject: Re: Web crawler on python

On Mon, Oct 27, 2008 at 6:54 AM, sonich <SoniaTerpugova at gmail.com> wrote:
> I need simple web crawler,
> I found Ruya, but it's seems not currently maintained.
> Does anybody know good web crawler on python or with python interface?

Simple, but  it works. Extend it all you like.

http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/e
xamples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -q, --quiet           Enable quiet mode
  -l, --links           Get links for specified url only
  -d DEPTH, --depth=DEPTH
                        Maximum depth to traverse

cheers
James

-- 
--
-- "Problems are solved by method"