Python FTP timeout value not effective

[David Bolen]

John Nagle <nagle at animats.com> writes:

> Here's the relevant code:
>
> TIMEOUTSECS = 60	## give up waiting for server after 60 seconds
> ...
> def urlopen(url,timeout=TIMEOUTSECS) :
>     if url.endswith(".gz") :	# gzipped file, must decompress first
>         nd = urllib2.urlopen(url,timeout=timeout)	# get connection
> 	... # (NOT .gz FILE, DOESN'T TAKE THIS PATH)
>     else :
> 	return(urllib2.urlopen(url,timeout=timeout)) # (OPEN FAILS)
>
>
> TIMEOUTSECS used to be 20 seconds, and I increased it to 60. It didn't
> help.

I apologize if it's an obvious question, but is there any possibility that
the default value to urlopen is not being used, but some other timeout is
being supplied?  Or that somehow TIMEOUTSECS is being redefined before
being used by the urlopen definition?  Can you (or have you) verified the
actual timeout parameter value being supplied to urllib2.urlopen?

The fact that you seem to still be timing out very close to the prior 20s
timeout value seems a little curious, since there's no timeout by default
(barring an application level global socket default), so it feels like a
value being supplied.

Not sure which 2.7 you're using, but I tried the below with both 2.7.3 and
2.7.5 on Linux since they were handy, and the timeout parameter seems to be
working properly at least in a case I can simulate (xxx is a firewalled
host so the connection attempt just gets black-holed until the timeout):

     >>> import time, urllib2
     >>> def test(timeout):
     ...   print time.ctime()
     ...   try:
     ...     urllib2.urlopen('ftp://xxx', timeout=timeout)
     ...   except:
     ...     print 'Error'
     ...   print time.ctime()
     ... 
     >>> test(5)
     Mon Sep  2 17:36:15 2013
     Error
     Mon Sep  2 17:36:20 2013
     >>> test(20)
     Mon Sep  2 17:36:23 2013
     Error
     Mon Sep  2 17:36:44 2013
     >>> test(60)
     Mon Sep  2 17:36:50 2013
     Error
     Mon Sep  2 17:37:50 2013

It's tougher to simulate a host that artificially delays the connection
attempt but then succeeds, so maybe it's an issue related specifically to
that implementation.  Depending on how the delay is implemented (delaying
SYN response versus accepting the connection but just delaying the welcome
banner, for example), I suppose it may be tickling some very specific bug.

Since all communication essentially boils down to I/O over the socket, it
seems to me likely that those cases should still fail over time periods
related to the timeout supplied, unlike your actual results, which makes me
wonder about the actual urllib2.urlopen timeout parameter.

-- David