[Tutor] fetching wikipedia articles

[amit sethi]

so is there a way around that problem ??

On Fri, Jan 23, 2009 at 2:25 PM, Andre Engels <andreengels at gmail.com> wrote:

> On Fri, Jan 23, 2009 at 9:09 AM, amit sethi <amit.pureenergy at gmail.com>
> wrote:
> > Well that is interesting but why should that happen in case  I am using a
> > different User Agent because I tried doing
> > status=rp.can_fetch('Mozilla/5.0',
> > "http://en.wikipedia.org/wiki/Sachin_Tendulkar")
> > but even that returns false
> > Is there something wrong with the syntax , Is there a catch that i don't
> > understand.
>
> The problem is that you are using the standard Python user agent when
> getting the robots.txt. Because the user agent is refused, it cannot
> get the robots.txt file itself to look at.
>
> --
> André Engels, andreengels at gmail.com
>



-- 
A-M-I-T S|S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20090123/3ff13138/attachment.htm>