[Python-Dev] Adding socket timeout to urllib2

[Guido van Rossum]

On 3/5/07, Facundo Batista <facundo at taniquetil.com.ar> wrote:
> I studied which modifications I need to make into urllib2 to support a
> socket timeout.
>
> - I'll modify urlopen for it to accept a socket_timeout parameter,
> default to None

I'd call it timeout. There can't really be much ambiguity can there?

> - Request will also accept a socket_timeout parameter, default to None.
> It will keep it in a socket_timeout attribute, so it could be
> added/modified from outside.

Ditto for the name.

> - OpenerDirector's 'open' method will accept socket_timeout, and build
> the Request object with it
>
> So, when it gets into the Handlers, how it'll be used? I checked the
> Handlers, and here is a resume of their 'open' or 'open'-like methods:
>
> - AbstractHTTPHandler: receives a class, and uses its .request and
> .getresponse methods
> - HTTPHandler: just calls AbstractHTTPHandler.do_open with
> httplib.HTTPConnection
> - HTTPSHandler: just calls AbstractHTTPHandler.do_open with
> httplib.HTTPSConnection
> - UnknownHandler: raises an Error
> - FileHandler: opens a file in a local host
> - FTPHandler: uses sockets in ftplib.FTP().connect, .login, .cwd, and
> .ntransfercmd, all through urllib's ftwrapper object
> - CacheFTPHandler: instantiates ftpwrapper object; does not uses it
> directly, though.
> - GopherHandler: is deprecated...
>
> So, these are the necessary further modifications in urllib2 beyond
> those detailed at the beggining of the mail:
>
> - Modify AbstractHTTPHandler to pass the socket_timeout parameter to
> the received class' methods.
> - Modify FTPHandler to pass the socket_timeout to ftpwrapper class.
> - Modify CacheFTPHandler the same way that FTPHandler
>
> Beyond that, I'll also need to:
>
> - Modify urllib.ftpwrapper to accept a socket_timeout parameter and
> pass it to ftplib.FTP object.
> - Add support to socket_timeout to ftplib's FTP object
> - Add support to socket_timeout to httplib's HTTPConnection and
> HTTPSConnection objects
>
> So, I think that in the name of clarity, I shall first modify ftplib
> and httplib, add socket_timeout where is necessary, and then add
> support to that in the more general urllib2.
>
> What do you think? Comments are very appreciated.

I say your next step should be to submit a patch, perhaps
incorporating the work that Skip has already done.

You could also reduce the amount of work to be done initially by only
handling http. At this point I expect that'll cover 99% of all uses.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)