Is Python good for web crawlers?

[Andrew Gwozdziewycz]

On 7 Feb 2006 08:33:28 -0800, Tempo <bradfordh at gmail.com> wrote:
> I was wondering if python is a good language to build a web crawler
> with? For example, to construct a program that will routinely search x
> amount of sites to check the availability of a product. Or to search
> for news articles containing the word 'XYZ'. These are just random
> ideas to try to explain my question a bit further. Well if you have an
> opinion about this please let me know becasue I am very interested to
> hear what you have to say. Thanks.
>

Google supplies a basic webcrawler as a google desktop plugin called
Kongulo (http://sourceforge.net/projects/goog-kongulo/) which is
written in python. I would think python would be perfect for this sort
of application. Your bottleneck is always going to be downloading the
page.

--
Andrew Gwozdziewycz <apgwoz at gmail.com>
http://ihadagreatview.org
http://plasticandroid.org