urllib2 request blocks

[John J. Lee]

Derek Fountain <devnull at example.com> writes:

> I just tried this:
> 
> >>> import urllib2
> >>> urllib2.urlopen( "https://passenger.ssc.com/~dmarti/contrib-faq/" )
> 
> It sits forever. Loading that URL in a normal browser gets me a popup asking
> about server certificate authenticity, which presumably is what urllib2 is
> blocking at.

That script doesn't hang for me, using Python 2.3.  I get a 200
response, with content the same as I see in Mozilla (Linux journal
author's guide and FAQ).  The socket module does not verify the
authenticity of servers, so neither does urllib or urllib2.


> How can I prepare my code for this eventuality and ensure it doesn't block?

Depends on why it's blocking.

sockets in 2.3 now support a timeout (that parameter is not fed
through to the httplib, urllib or urllib2 public interfaces, but you
can still set the default socket timeout in your application code).
If it's DNS that blocking, though, that won't help you.  I think
proxies like squid do DNS lookups in a separate process (I guess you
can use threads too).

Alternatively, you can start messing with things like asyncore or
Twisted, or use threads (I don't recommend the latter :-/).


John