[Tutor] fetching wikipedia articles

[Andre Engels]

On Fri, Jan 23, 2009 at 11:25 AM, Andre Engels <andreengels at gmail.com> wrote:
> On Fri, Jan 23, 2009 at 10:37 AM, amit sethi <amit.pureenergy at gmail.com> wrote:
>> so is there a way around that problem ??
>
> Ok, I have done some checking around, and it seems that the Wikipedia
> server is giving a return code of 403 (forbidden), but still giving
> the page - which I think is weird behaviour. I will check with the
> developers of Wikimedia why this is done,

It appears that this is done on purpose, not just for Python but also
for the 'standard' user agent in other languages. The idea is that it
forces programmers to add their own user agent, so that if the program
trying to contact Wikipedia misbehaves, it can be blocked or otherwise
handeled with; as a bonus it also gives programmers a small extra
hurdle so that the most amateuristic attempts are stopped, but more
thought out programs are not.


-- 
André Engels, andreengels at gmail.com