Python FTP timeout value not effective

John Nagle nagle at animats.com
Mon Sep 2 13:43:19 EDT 2013


    I'm reading files from an FTP server at the U.S. Securities and
Exchange Commission.  This code has been running successfully for
years.  Recently, they imposed a consistent connection delay
of 20 seconds at FTP connection, presumably because they're having
some denial of service attack.  Python 2.7 urllib2 doesn't
seem to use the timeout specified.  After 20 seconds, it
gives up and times out.

Here's the traceback:

Internal error in EDGAR update: <urlopen error ftp error: [Errno 110]
Connection timed out>
....
  File "./edgar/edgarnetutil.py", line 53, in urlopen
    return(urllib2.urlopen(url,timeout=timeout))
  File "/opt/python27/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/opt/python27/lib/python2.7/urllib2.py", line 394, in open
    response = self._open(req, data)
  File "/opt/python27/lib/python2.7/urllib2.py", line 412, in _open
    '_open', req)
  File "/opt/python27/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/opt/python27/lib/python2.7/urllib2.py", line 1379, in ftp_open
    fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout)
  File "/opt/python27/lib/python2.7/urllib2.py", line 1400, in connect_ftp
    fw = ftpwrapper(user, passwd, host, port, dirs, timeout)
  File "/opt/python27/lib/python2.7/urllib.py", line 860, in __init__
    self.init()
  File "/opt/python27/lib/python2.7/urllib.py", line 866, in init
    self.ftp.connect(self.host, self.port, self.timeout)
  File "/opt/python27/lib/python2.7/ftplib.py", line 132, in connect
    self.sock = socket.create_connection((self.host, self.port),
self.timeout)
  File "/opt/python27/lib/python2.7/socket.py", line 571, in
create_connection
    raise err
URLError: <urlopen error ftp error: [Errno 110] Connection timed out>

Periodic update completed in 21.1 seconds.
----------------------------------------------

Here's the relevant code:

TIMEOUTSECS = 60	## give up waiting for server after 60 seconds
...
def urlopen(url,timeout=TIMEOUTSECS) :
    if url.endswith(".gz") :	# gzipped file, must decompress first
        nd = urllib2.urlopen(url,timeout=timeout)	# get connection
	... # (NOT .gz FILE, DOESN'T TAKE THIS PATH)
    else :
	return(urllib2.urlopen(url,timeout=timeout)) # (OPEN FAILS)


TIMEOUTSECS used to be 20 seconds, and I increased it to 60. It didn't
help.

This isn't an OS problem. The above traceback was on a Linux system.
On Windows 7, it fails with

"URLError: <urlopen error ftp error: [Errno 10060] A connection attempt
failed because the connected party did not properly respond after a
period of time, or established connection failed because connected host
has failed to respond>"

But in both cases, the command line FTP client will work, after a
consistent 20 second delay before the login prompt.  So the
Python timeout parameter isn't working.

				John Nagle






More information about the Python-list mailing list