From mal at egenix.com Mon Mar 15 21:55:08 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 15 Mar 2010 21:55:08 +0100 Subject: [pyOpenSSL] Problem with pyOpenSSL 0.10 and OpenSSL 0.9.8m In-Reply-To: <20100226162625.2792.410394570.divmod.xquotient.109@localhost.localdomain> References: <4B87B663.6010600@egenix.com> <4B87B8CD.1060109@egenix.com> <20100226162625.2792.410394570.divmod.xquotient.109@localhost.localdomain> Message-ID: <4B9E9EAC.8010703@egenix.com> exarkun at twistedmatrix.com wrote: > On 12:04 pm, mal at egenix.com wrote: >> M.-A. Lemburg wrote: >>> OpenSSL just released version 0.9.8m where they changed the >>> SSL renegotiation scheme to now follow RFC 5746 instead of >>> just disabling it completely: >>> >>> http://tools.ietf.org/html/rfc5746 >>> >>> pyOpenSSL compiles against the new version without problems, >>> but one of the unit tests fails on Linux x64 (and perhaps >>> other platforms as well): >>>> python OpenSSL/test/test_ssl.py >>> .......E.... >>> ====================================================================== >>> ERROR: test_set_default_verify_paths (__main__.ContextTests) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> File "OpenSSL/test/test_ssl.py", line 253, in >>> test_set_default_verify_paths >>> clientSSL.do_handshake() >>> Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate >>> verify failed')] >>> >>> ---------------------------------------------------------------------- >>> Ran 12 tests in 0.660s >>> >>> FAILED (errors=1) >>> >>> The test connects to https://verisign.com/ >>> I've tried a few other sites as well, but always get the same >>> error. >> >> I tried the same on Linux x86 with the same results. >> >> I then also checked pyOpenSSL 0.9.0 with OpenSSL 0.9.8l >> and again get the same results. >> >> Perhaps there's something wrong with my test setup ? I've >> done the tests on two different machines. > > This test depends on OpenSSL having been built so that it has access to > the platform-provided CA certificate database. It sounds like your > builds weren't done this way. > > I've never actually built OpenSSL this way myself, and I have very > little idea what is involved in doing so. I know that Ubuntu's OpenSSL > builds have this enabled, but as far as I know none of the other widely > used builds for other platforms do (I've looked at OS X and Windows and > they don't, I'm not sure about other Linux distros). > > Unfortunately I don't have much more info than that about the feature, > so I can't make any suggestions about how to check to see if this is > really the problem, or how to change the build in order to fix it. Thanks for the hint. Here's the full background: http://www.madboa.com/geek/openssl/#verify-system Different OSes place the trusted certificate database files in different places and non of them uses the OpenSSL default which is /usr/local/ssl and a subdir certs/. The only way to work around this appears to be to call ctx.load_verify_locations() to point the context to the right set of trusted certificates. I believe that the test should apply such a setup for the verisign.com certificate authority instead of relying on a platform provided default setup, ie. use its own certs/ subdir with the root CA certificates that are used by verisign.com. In any case, the above test failure is a problem with the test setup more than anything else. Here's a version of the test which works on OpenSUSE: # Arg, verisign.com doesn't speak TLSv1 context = Context(SSLv3_METHOD) #context.set_default_verify_paths() context.load_verify_locations(None, '/etc/ssl/certs') context.set_verify( VERIFY_PEER, lambda conn, cert, errno, depth, preverify_ok: preverify_ok) client = socket() client.connect(('verisign.com', 443)) clientSSL = Connection(context, client) clientSSL.set_connect_state() clientSSL.do_handshake() clientSSL.send('GET / HTTP/1.0\r\n\r\n') self.assertTrue(clientSSL.recv(1024)) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 15 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From exarkun at twistedmatrix.com Mon Mar 15 22:35:33 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 15 Mar 2010 21:35:33 -0000 Subject: [pyOpenSSL] Problem with pyOpenSSL 0.10 and OpenSSL 0.9.8m In-Reply-To: <4B9E9EAC.8010703@egenix.com> References: <4B87B663.6010600@egenix.com> <4B87B8CD.1060109@egenix.com> <20100226162625.2792.410394570.divmod.xquotient.109@localhost.localdomain> <4B9E9EAC.8010703@egenix.com> Message-ID: <20100315213533.2792.2006907500.divmod.xquotient.1645@localhost.localdomain> On 08:55 pm, mal at egenix.com wrote: > >Thanks for the hint. Here's the full background: > >http://www.madboa.com/geek/openssl/#verify-system Awesome link, thanks for finding it. >Different OSes place the trusted certificate database files in >different places and non of them uses the OpenSSL default which >is /usr/local/ssl and a subdir certs/. The version of OpenSSL distributed by Ubuntu and Debian and by Apple in OS X 10.6 don't use the default, but the build configuration is correctly specified so that OpenSSL can find the certificates in the non-default location. In a perfect world, this is how it would be on all platforms. :) > >The only way to work around this appears to be to call >ctx.load_verify_locations() to point the context to the >right set of trusted certificates. > >I believe that the test should apply such a setup for >the verisign.com certificate authority instead of >relying on a platform provided default setup, ie. use >its own certs/ subdir with the root CA certificates that >are used by verisign.com. The test in question is for ctx.set_default_verify_paths. It does seem very likely that calling ctx.load_verify_locations with a path known to contain the necessary CA certificate would make the test pass, but then it would be testing something else. :) I'll definitely agree that this is not a very good unit test, for a number of reasons. > >In any case, the above test failure is a problem with the test >setup more than anything else. I agree. I think it might make sense to try to determine if the libssl being used was built in a way that could allow the necessary CA cert to be found, and skip the test if it was not. Running "openssl version -d" and checking what's in that directory might be a good heuristic. > >Here's a version of the test which works on OpenSUSE: > >[snip] Any chance you could agitate to have the OpenSUSE build fixed so that ctx.set_default_verify_paths can actually work? :) If all the major platforms had a properly built version of OpenSSL, I think it'd be a lot easier for people to write well-behaved SSL applications. Jean-Paul From mal at egenix.com Tue Mar 16 12:56:10 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 16 Mar 2010 12:56:10 +0100 Subject: [pyOpenSSL] Problem with pyOpenSSL 0.10 and OpenSSL 0.9.8m In-Reply-To: <20100315213533.2792.2006907500.divmod.xquotient.1645@localhost.localdomain> References: <4B87B663.6010600@egenix.com> <4B87B8CD.1060109@egenix.com> <20100226162625.2792.410394570.divmod.xquotient.109@localhost.localdomain> <4B9E9EAC.8010703@egenix.com> <20100315213533.2792.2006907500.divmod.xquotient.1645@localhost.localdomain> Message-ID: <4B9F71DA.8000000@egenix.com> exarkun at twistedmatrix.com wrote: > On 08:55 pm, mal at egenix.com wrote: >> >> Thanks for the hint. Here's the full background: >> >> http://www.madboa.com/geek/openssl/#verify-system > > Awesome link, thanks for finding it. > >> Different OSes place the trusted certificate database files in >> different places and non of them uses the OpenSSL default which >> is /usr/local/ssl and a subdir certs/. > > The version of OpenSSL distributed by Ubuntu and Debian and by Apple in > OS X 10.6 don't use the default, but the build configuration is > correctly specified so that OpenSSL can find the certificates in the > non-default location. > > In a perfect world, this is how it would be on all platforms. :) And it probably is on most platforms... I was testing with our embedded OpenSSL solution which uses the OpenSSL build defaults. Since most OSes appear to use non-standard certificate database locations, there's little hope of relying on the default verify path compiled into OpenSSL in such a case. >> The only way to work around this appears to be to call >> ctx.load_verify_locations() to point the context to the >> right set of trusted certificates. >> >> I believe that the test should apply such a setup for >> the verisign.com certificate authority instead of >> relying on a platform provided default setup, ie. use >> its own certs/ subdir with the root CA certificates that >> are used by verisign.com. > > The test in question is for ctx.set_default_verify_paths. It does seem > very likely that calling ctx.load_verify_locations with a path known to > contain the necessary CA certificate would make the test pass, but then > it would be testing something else. :) True. I missed that point :-) > I'll definitely agree that this is not a very good unit test, for a > number of reasons. I guess using that API with our pyOpenSSL distribution is not really all that useful. OTOH, explicit is better than implicit and IMHO it's better to explicitly have the user or application define which root CAs to trust or not. We'll add a note to the web page. >> In any case, the above test failure is a problem with the test >> setup more than anything else. > > I agree. I think it might make sense to try to determine if the libssl > being used was built in a way that could allow the necessary CA cert to > be found, and skip the test if it was not. Running "openssl version -d" > and checking what's in that directory might be a good heuristic. That won't work with our embedded version, since we don't ship the openssl executable, only the libs. Too bad, that there's not SSL_CTX_get_default_verify_paths(). >> Here's a version of the test which works on OpenSUSE: >> >> [snip] > > Any chance you could agitate to have the OpenSUSE build fixed so that > ctx.set_default_verify_paths can actually work? :) > > If all the major platforms had a properly built version of OpenSSL, I > think it'd be a lot easier for people to write well-behaved SSL > applications. OpenSUSE's OpenSSL libs do have the paths correclty defined. It's just that those OS libs often tend to be rather outdated and, in some cases, also have (had) serious bugs. That's the reason why we've created the eGenix pyOpenSSL distribution - it includes the OpenSSL libs embedded in the OpenSSL package and does not rely on the system provided ones. A side effect of this, is that the default OPENSSLDIR is (currently) set to /usr/local/ssl for those libs, so it's rather unlikely that the libs find the OS provided default collection of trusted root CA certs. Perhaps we should change the default to either /dev/null to make this even more secure in the "explicit is better than implicit" sense or add a function which tries to determine the OS default location for the root CA certs in order to help the developer in adding support for trusting these. Our own use case, the mxODBC Connect product, doesn't need all this, since you have to explicitly define the trusted certificates anyway. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 16 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/