waiting for html to load: a followup

[John J. Lee]

Josh <joshl at commenspace.org> writes:
[...]
> Anyway, I decided to forget IE and I am now trying to use urllib2 to
> open up the page, read it, etc.  My problem is the page has a built-in
> refresh and I don't know how to have python re-read the page until
> it's ready to hand over the links.
> 
> An example of the page is:
> http://edcw2ks23.cr.usgs.gov/Website/zipship/waiting.jsp?areaList=49.0,47.0,-122.0,-124.08&prodList=NED,

Example, with some debugging turned on so you can see some of what's
going on:

import ClientCookie
opener = ClientCookie.build_opener(
    ClientCookie.HTTPRefreshProcessor(max_time=None),
    ClientCookie.HTTPResponseDebugProcessor(),
    ClientCookie.HTTPRedirectDebugProcessor(),
    )
ClientCookie.getLogger("ClientCookie").setLevel(ClientCookie.DEBUG)

r = opener.open('http://edcw2ks23.cr.usgs.gov/Website/zipship/waiting.jsp?areaList=49.0,47.0,-122.0,-124.08&prodList=NED,')
f = open('out.html', 'w')
f.write(r.read())


Don't mix ClientCookie and urllib2, BTW.


John