Merlin, a fun little program

[Oren Tirosh]

On Mon, Jul 07, 2003 at 02:05:28PM -0700, Ron Stephens wrote:
> I guess the bigger question is; is there anything wrong with web
> scraping? I surely never meant any harm in it, and certainly no money
> is involved. But maybe I should give it up and do other things?

I don't think there's anything fundamentally wrong with web scraping but
you have to consider the fact that a single script can easily consume 
resources that cost real money and would otherwise serve thousands of 
human users. If a provider installs mechanisms to detect scripts and block 
them this can quickly become a cat-and-mouse game where the scrapers try 
to fool these mechanisms, it starts to get ugly and everybody suffers.

I think Google handled this very well - defusing most of the problem by
letting people have what they want while keeping things under control. 
I generally find the way Google handles the issues that come with their 
dominant market position quite "Pythonic". 

    Oren