Question about using urllib2 to load a url

[John J. Lee]

"Kushal Kumaran" <kushal.kumaran at gmail.com> writes:
[...]
> If, at any time, an error response fails to reach your machine, the
> code will have to wait for a timeout.  It should not have to wait
> forever.
[...]

...but it might have to wait a long time.  Even if you use
socket.setdefaulttimeout(), DNS lookups can block for a long time.
The way around that is to use Python threads (no need to try to "kill"
the thread that's doing the urlopen() -- just ignore that thread if it
takes too long to finish).

Looks like 2.6 will have socket timeouts exposed at the urllib2 level
(so no need to call socket.setdefaulttimeout() any more), but the need
to use threads with urllib2 to get timeouts will remain in many cases,
due to the DNS thing (the same applies to urllib, or any other module
that ends up doing DNS lookups).


John