Deficiency in urllib/socket for https?

[John J. Lee]

jjl at pobox.com (John J. Lee) writes:

> jjl at pobox.com (John J. Lee) writes:
> [...]
> > Would you mind submitting a doc patch (both urllib and urllib2 docs
> > appear to need fixing -- urllib2 to say that it never verifies, urllib
> > to say that it skips verification if an appropriate x509 mapping isn't
> > supplied)?
> 
> Hmm, maybe I've got this wrong: the fact that key/cert args are passed
> to httplib.HTTPS by urllib doesn't mean authentication happens, and
> the fact that they're not passed by urllib2 doesn't mean
> authentication doesn't happen.  I'll investigate.

Bah!  *After* reading the source, I found this in the ssl module docs:

| Warning: This does not do any certificate verification!

(which the _ssl.c source confirms: it uses SSL_VERIFY_NONE, but
doesn't call SSL_get_verify_result).

So the urllib docs are wrong:

| Additional keyword parameters, collected in x509, are used for
| authentication with the https: scheme. The keywords key_file and
| cert_file are supported; both are needed to actually retrieve a
| resource at an https: URL.

They're not needed, and they're never used for authentication (if you
don't count just checking the key without verifying it against the
certificate).  Given this, the fact that urllib2 doesn't have
arguments for this starts to look like a feature, not a bug!  Actually
(dredging up very hazy memories here) aren't you supposed to check a
revocation list, too?  Is that given in a URL in the certificate?  No
idea how this SSL stuff is supposed to work, really...

I'll upload a doc patch in a minute.

So, in summary, none of httplib, urllib and urllib2 in standard Python
do proper authentication (because the socket module doesn't).  There
are third-party SSL libraries for Python: m2crypto is one.  If you
need it, and assuming m2crypto has an ssl function with the same
interface that *does* do better auth, I suppose you could probably do

import socket
from m2crypto import ssl  # or whatever
socket.ssl = ssl


And have urllib magically start working, with any luck.


John