[Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)

[Nick Coghlan]

On 26 November 2015 at 06:17, Robert Collins <robertc at robertcollins.net> wrote:
> On 26 November 2015 at 08:57, Barry Warsaw <barry at python.org> wrote:
>> There's a lot to process in this thread, but as I see it, the issue breaks
>> down to these questions:
>>
>> * How should PEP 493 be implemented?
>>
>> * What should the default be?
>>
>> * How should PEP 493 be worded to express the right tone to redistributors?
>>
>> Let me take on the implementation details here.
>>
>> On Nov 24, 2015, at 04:04 PM, M.-A. Lemburg wrote:
>>
>>>I would still find having built-in support for the recommendations
>>>in the Python stdlib a better approach
>>
>> As would I.
>
> For what its worth: a PEP telling distributors to patch the standard
> library is really distasteful to me.

I'm not a big fan of it either, but it's the way sustainable
commercial open source distribution works in practice: customers want
selective upgrades, so customers get selective upgrades; customers
want greater platform integration, so customers get greater platform
integration; etc, etc. Large Python shops like Google will typically
maintain their own custom Python variants with their own patches
applied that are suitable and sufficient for their use cases, but may
not be suitable for pushing upstream (e.g. they may be Linux specific
patches, without support for other platforms).

That's why I've focused on providing guidance to redistributors to
keep their patches within reasonably consistent bounds for PEPs like
bundling pip, backporting the network security enhancements to Python
2.7, and changing the default behaviour for HTTPS verification.
Different redistributors have different customer bases, which also
differ from the audience for upstream python.org releases, so saying
"don't patch Python" is denying the reality of the obligations
commercial vendors have to their customers, while "when you patch
Python, please ensure you abide by these guidelines" is something
redistributors can realistically do.

> We've spent a long time trying to build close relations such that when
> something doesn't work distributors can share their needs with us and
> we can make Python out of the box be a good fit. This seems to fly in
> the exact opposite direction: we're explicitly making it so that
> Python builds on these vendor's platforms will not be the same as you
> get by checking out the Python source code.

This has always been true for the RHEL system Python - it gets
backported changes, just like the kernel does (albeit nowhere near as
many). That's one of the main things long term support customers are
paying for - developers continuing to care about a code base after the
volunteers working on the project upstream have (entirely reasonably)
shifted their attentions to newer versions. We try to keep the patches
to a minimum, and avoid having to carry large patches indefinitely by
only backporting changes that have already been accepted in later
versions, but "never apply patches to upstream projects as a
downstream redistributor" isn't a realistic goal.

In this particular case, the migration problems were already raised in
the PEP 476 discussions, and the decision was made to *not* provide a
public API specifically for globally turning off HTTPS certificate
verification, since that would either:

1. Need to be added to Python 3 and maintained indefinitely; or
2. Be implemented in Python 2.7 only, and thus create another barrier
to 2->3 migration

A HTTPS specific flag wasn't considered a suitable long term API
design for Python 3, while a multi-protocol configuration design that
also handled setting the defaults for other protocols like secure
SMTP, IMAP, POP3, SFTP, etc seemed generally desirable (but also
doesn't exist anywhere, and poses signficant challenges in designing
and implementing a good test matrix).

The only concrete things that have changed since PEP 476 was approved is that:

* we figured out the monkeypatching based migration recommendations in
that PEP didn't actually work for the redistributor case
* Robert Kuska came up with a way to do a backwards compatible
backport of PEP 476 to RHEL 7.2 that doesn't raise any new barriers to
migrating from Python 2 to Python 3

PEP 493 originally only described the latter idea - I added MAL's
design for the environment variable based security downgrade based on
the original PEP 493 discussions in July. Splitting the latter out to
its own PEP that proposes it as a forwards compatible configuration
setting akin to "curl --insecure" and "wget --no-check-certificate"
for Python 2.7.12, 3.4.4, 3.5.2 and 3.6 upstream would be fine by me,
since that would also significantly clarify the fact that the two
ideas are entirely orthogonal to each other (except for the need to
define a relative priority when you combine the two).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia