urllib2 hangs "forever" where there is no network interface

[John J. Lee]

(I'm having news trouble, sorry if anybody sees a similar reply three
times...)

"dumbkiwi" <dmbkiwi at gmail.com> writes:
> On Feb 2, 5:02 am, j... at pobox.com (John J. Lee) wrote:
> > "dumbkiwi" <dmbk... at gmail.com> writes:
[...]
> > > If there is no network interface, urllib2 hangs for a very long time
> > > before it raises an exception.  I have set the socket timeout with
> > > socket.setdefaulttimeout(), however, where there is no network
> > > interface, this seems to be ignored - presumably, because without a
> > > network interface, there is nothing for the socket module to interact
> > > with.
[...]
> > Presumably Windows?  (Unix systems almost always have at least a
> > loopback interface)
> >
> > John
> 
> Sorry, I should have been more specific.  The network interfaces are
> up - ie lo and eth1, it's where the wireless connection has dropped
> out.

The underlying problem is that Python's socket timeout is implemented
using select() or poll().  Those system calls only allow timing out
activity on file descriptors (e.g. sockets).  The problem you're
seeing is caused by getaddrinfo() blocking for a long time, and that
function doesn't involve file descriptors.  The problem should really
be fixed at the C level (in Modules/socketmodule.c), using something
like alarm() or a thread to apply a timeout to getaddrinfo() calls.


> Is the best solution to test for a wireless connection through /
> proc before trying to download data?

That may be a good practical solution.

Another workaround that might be useful is to do your DNS lookups only
once, then use only IP addresses.


John