Is Python good for web crawlers?

[Magnus Lycka]

Tempo wrote:
> I was wondering if python is a good language to build a web crawler
> with? For example, to construct a program that will routinely search x
> amount of sites to check the availability of a product. Or to search
> for news articles containing the word 'XYZ'. These are just random
> ideas to try to explain my question a bit further. Well if you have an
> opinion about this please let me know becasue I am very interested to
> hear what you have to say. Thanks.

I dunno, but there are these two guys, Sergey Brin and Lawrence Page,
who wrote a web crawler in Python. As far as I understood, they were
fairly successful with it. I think they called their system Koogle,
Bugle, or Gobble or something like that. Goo...can't remember.

See http://www-db.stanford.edu/~backrub/google.html

They've also employed some clever Python programmers, such as Greg
Stein, Alex Martelli (isn't he a bot?) and some obscure dutch
mathematician called Guido van something. It seems they still like
Python.