[Python-Dev] SSL certificates recommendations for downstream python packagers

[Christian Heimes]

On 2017-01-30 14:53, David Cournapeau wrote:
> Hi,
> 
> I am managing the team responsible for providing python packaging at
> Enthought, and I would like to make sure we are providing a good (and
> secure) out of the box experience for SSL.
> 
> My understanding is that PEP 476 is the latest PEP that concerns this
> issue, and that PEP recommends using the system store:
> https://www.python.org/dev/peps/pep-0476/#trust-database. But looking at
> binary python distributions from python.org <http://python.org>, that
> does not seem to a.ways be the case. I looked at the following:
> 
> * 3.5.3 from python.org <http://python.org> for OS X (64 bits): this
> uses the old, system openssl

On macOS, system OpenSSL uses TEA to look up trust anchors from the
keyring, https://hynek.me/articles/apple-openssl-verification-surprises/

> * 3.6.0 from python.org <http://python.org> for OS X: this embeds a
> recent openssl, but ssl seems to be configured to use non existing paths
> (ssl..get_default_verify_paths()), and indeed, cert validation seems to
> fail by default with those installers

I guess you haven't read the warning on the download page,
https://www.python.org/downloads/release/python-360/ :)

tl;dr Python on macOS can no longer use the keyring for certificates.
You have to install certifi. Personally I (as a maintainer of the ssl
module) do neither like certifi nor the approach taken. It's wrong and
dangerous because certifi by-passes system policies and does not include
local CAs. But it's the best we got.

> * 3.6.0 from python.org <http://python.org> for windows: I have not
> found how the ssl module finds the certificate, but certification seems
> to work

The magic happens in load_default_certs(),
https://hg.python.org/cpython/file/tip/Lib/ssl.py#l445 . The approach
does work most of the time. Sometimes it fails because Windows download
root CA's on demand. The code does not trigger a download.

Steve and I have some ideas to improve the situation,
https://bugs.python.org/issue28747 .

> 
> Are there any official recommendations for downstream packagers beyond
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
> default cert locations ?

My semi-official recommendations are:

* On Linux and BSD, use OpenSSL from your vendor (Red Hat, SuSE, Ubuntu,
Debian, Arch, Gentoo, ...). Vendors compile OpenSSL with special flags
to make get_default_verify_paths() work.

* If your vendor prefers LibreSSL, you might run into minor issue.
LibreSSL is not fully compatible to OpenSSL. Your vendor also compiles
LibresSL with special flags.

* Windows kinda works. We are going to improve the situation for 3.7.

* On macOS ... well, please contact Apple service and demand a solution.
Cory Benfield's TLS ABC PEP and Paul Kehrer's work on SecureTransport
bindings for Python will let you use Apple's native TLS lib (with some
restrictions regarding fork()). In the mean time you are screwed and
have to rely on certifi.

As last resort, use certifi. You HAVE to make sure to keep it up to date
and you have to deal with local policies and CAs yourself.

Christian