[Baypiggies] urllib2.urlopen() and exception layering

[Niall O'Higgins]

Hi Gregory,

On Sat, Apr 05, 2008 at 11:54:32PM -0700, Gregory P. Smith wrote:
> urllib2 is a pretty big mess and does unfortunately violate the general rule
> of exception catching + retyping + reraising being a bad idea.
>
> That said, you can reduce the above list of exceptions a bit thanks to their
> inheritance.  URLError inherits from IOError.  All httplib errors inherit
> from httplib.HTTPException.  This leaves you with (IOError, socket.error,
> httplib.HTTPException, ValueError) as the ones you should catch.  I changed
> socket.error to inherit from IOError in Python 2.6.  That narrows the list
> down to a sane pair of exceptions if the ValueErrors can be fixed.
>
> IMHO if you're ever seeing a ValueError out of urllib2 in normal sane use
> please try to figure out why, make a simple test case and file it on
> bugs.python.org if you believe it is a case that the library should handle.
> double plus good if it includes a patch to fix it.

Thanks for the informative reply.  I have modified my long-running
HTTP fetch programs to track down this ValueError condition, hopefully
it will crop up again one of these days.

Apart from modifying the code, should the above comments be mentioned
in the standard library docs?  It seemed unusual to me that only
URLError was explicitly mentioned in the urllib2.urlopen()
description.  I will write a path to the documentation if so,
mentioning the various exceptions which can be thrown.  I'm just not
sure how Python documentation is managed or maintained.

Cheers!

>
>
> >        try:
> >            h.request(req.get_method(), req.get_selector(), req.data,
> > headers)
> >            r = h.getresponse()
> >        except socket.error, err: # XXX what error?
> >            raise URLError(err)
> >
> > I think this is a problem which needs to be addressed, at the very least
> > through clearer documentation, and possibly by improving urllib2 to handle
> > more
> > of these exceptions and raise them as URLError.
> >
> > I'm new to the Python development community, but would be happy to submit
> > a
> > patch if there is some consensus on which approach to take.
> >
> > Thanks!
> >

--
Niall O'Higgins
Software Enthusiast
http://niallohiggins.com