mysteries of urllib/urllib2

[Adrian Smith]

On Jul 4, 12:42 am, Ben Cartwright <benc.nos... at gmail.com> wrote:
> On Jul 3, 11:14 am, Adrian Smith <adrian_p_sm... at yahoo.com> wrote:
>
> > > > The following (pinched
> > > > from Dive Into Python) seems to work perfectly in Idle, but
> > > > falls at the final hurdle when run as a cgi script
> > > Put this at the top of your cgi script:
>
> > > import cgitb; cgitb.enable()
>
> Did you even try this?  Asking for Python help without posting the
> traceback is like phoning your mechanic and saying, "My car is
> making a generic rattling noise, can you tell me what the problem
> is without looking under the hood?"

Sorry, I thought as the cgi did appear to have web access it wasn't
applicable, and it's amazing what some mechanics can infer from engine
noise. cgitb certainly does send back an impressive amount of
information, I'll be sure to use it in future.

> > Apparently there's a way to change the user-agent string
> > by subclassing urllib's URLopener class, but that's beyond my
> > comfort zone at present.
>
> Untested:
>
> import urllib
> url = 'http://groups.google.com/group/Google-AJAX-Search-API/
> browse_thread/thread/a0eb87ad13b11762'
> opener = urllib.FancyURLopener()
> opener.addheaders = [('User-Agent', 'Fauxzilla 4.0')]
> data = opener.open(url).read()

That works a treat, thanks!