[Python-Dev] Request for pronouncement on PEP 493 (HTTPS verification backport guidance)
Wes Turner
wes.turner at gmail.com
Mon Nov 23 15:51:01 EST 2015
... Just had this discussion in regards to easy_install, Ubuntu 14.04 LTS,
and the ReadTheDocs Docker images (as well as: ~why should I have to
wget/curl get-pip.py)
https://github.com/rtfd/readthedocs-docker-images/pull/3
On Nov 23, 2015 2:47 PM, "Wes Turner" <wes.turner at gmail.com> wrote:
> 1. Does this affect easy_install?
> 2. If/because this affects easy_install,
> should the guidance / suggested package installation tool be [pip];
> because pip install_requires backports.ssl_match_hostname
>
> https://pypi.python.org/pypi/backports.ssl_match_hostname
> On Nov 10, 2015 6:48 PM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
>> Hi folks,
>>
>> I have a confession to make - I dropped the ball on the HTTPS
>> verification backport proposals in PEP 493, and let the upstream and
>> downstream approval processes get out of sequence.
>>
>> As a result, the RHEL 7.2 beta released back in September incorporates
>> the HTTPS verification feature backport based on the current PEP 493
>> draft, even though that hasn't formally been pronounced as an Active
>> recommendation by python-dev yet.
>>
>> Accordingly, I'm belatedly submitting it for pronouncement now:
>> https://www.python.org/dev/peps/pep-0493/
>>
>> There's currently no BDFL-Delegate assigned, so if Guido doesn't want
>> to handle it, we'll need to address that question first.
>>
>> Our last discussion back in July seemed to show that folks either
>> didn't care about the question (because they're using unmodified
>> upstream versions so the PEP didn't affect them), or else thought the
>> approach described in the PEP was reasonable, so I'm hoping the
>> consequences of my mistake won't be too severe.
>>
>> Regards,
>> Nick.
>>
>> P.S. I'm aware that this looks like presenting a fait accompli at a
>> point where it's too late to realistically say "No", but the truth is
>> that preparation for the Python in Education miniconf at PyCon
>> Australia ramped up immediately after the July discussion, and then I
>> personally got confused as to the scope of what was being included in
>> 7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
>> being deferred to a later release, but it's actually the whole package
>> of 466+476+493). That's my fault for trying to keep track of too many
>> things at once (and thus failing at some of them), not anyone else's.
>>
>> ================================
>>
>> PEP: 493
>> Title: HTTPS verification recommendations for Python 2.7 redistributors
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Nick Coghlan <ncoghlan at gmail.com>,
>> Robert Kuska <rkuska at redhat.com>,
>> Marc-André Lemburg <mal at lemburg.com>
>> Status: Draft
>> Type: Informational
>> Content-Type: text/x-rst
>> Created: 10-May-2015
>> Post-History: 06-Jul-2015
>>
>>
>> Abstract
>> ========
>>
>> PEP 476 updated Python's default handling of HTTPS certificates to be
>> appropriate for communication over the public internet. The Python 2.7
>> long
>> term maintenance series was judged to be in scope for this change, with
>> the
>> new behaviour introduced in the Python 2.7.9 maintenance release.
>>
>> This PEP provides recommendations to downstream redistributors wishing to
>> provide a smoother migration experience when helping their users to manage
>> this change in Python's default behaviour.
>>
>>
>> Rationale
>> =========
>>
>> PEP 476 changed Python's default behaviour to better match the needs and
>> expectations of developers operating over the public internet, a category
>> which appears to include most new Python developers. It is the position of
>> the authors of this PEP that this was a correct decision.
>>
>> However, it is also the case that this change *does* cause problems for
>> infrastructure administrators operating private intranets that rely on
>> self-signed certificates, or otherwise encounter problems with the new
>> default
>> certificate verification settings.
>>
>> The long term answer for such environments is to update their internal
>> certificate management to at least match the standards set by the public
>> internet, but in the meantime, it is desirable to offer these
>> administrators
>> a way to continue receiving maintenance updates to the Python 2.7 series,
>> without having to gate that on upgrades to their certificate management
>> infrastructure.
>>
>> PEP 476 did attempt to address this question, by covering how to revert
>> the
>> new settings process wide by monkeypatching the ``ssl`` module to restore
>> the
>> old behaviour. Unfortunately, the ``sitecustomize.py`` based technique
>> proposed
>> to allow system administrators to disable the feature by default in their
>> Standard Operating Environment definition has been determined to be
>> insufficient in at least some cases. The specific case of interest to the
>> authors of this PEP is the one where a Linux distributor aims to provide
>> their users with a
>> `smoother migration path
>> <https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
>> than the standard one provided by consuming upstream CPython 2.7 releases
>> directly, but other potential challenges have also been pointed out with
>> updating embedded Python runtimes and other user level installations of
>> Python.
>>
>> Rather than allowing a plethora of mutually incompatibile migration
>> techniques
>> to bloom, this PEP proposes two alternative approaches that redistributors
>> may take when addressing these problems. Redistributors may choose to
>> implement
>> one, both, or neither of these approaches based on their assessment of the
>> needs of their particular userbase.
>>
>> These designs are being proposed as a recommendation for redistributors,
>> rather
>> than as new upstream features, as they are needed purely to support legacy
>> environments migrating from older versions of Python 2.7. Neither approach
>> is being proposed as an upstream Python 2.7 feature, nor as a feature in
>> any
>> version of Python 3 (whether published directly by the Python Software
>> Foundation or by a redistributor).
>>
>>
>> Requirements for capability detection
>> =====================================
>>
>> As these recommendations are intended to cover backports to earlier Python
>> versions, the Python version number cannot be used as a reliable means for
>> detecting them. Instead, the recommendations are defined to allow the
>> presence
>> or absence of the feature to be determined using the following technique::
>>
>> python -c "import ssl; ssl._relevant_attribute"
>>
>> This will fail with `AttributeError` (and hence a non-zero return code)
>> if the
>> relevant capability is not available.
>>
>> The marker attributes are prefixed with an underscore to indicate the
>> implementation dependent nature of these capabilities - not all Python
>> distributions will offer them, only those that are providing a multi-stage
>> migration process from the legacy HTTPS handling to the new default
>> behaviour.
>>
>>
>> Recommendation for an environment variable based security downgrade
>> ===================================================================
>>
>> Some redistributors may wish to provide a per-application option to
>> disable
>> certificate verification in selected applications that run on or embed
>> CPython
>> without needing to modify the application itself.
>>
>> In these cases, a configuration mechanism is needed that provides:
>>
>> * an opt-out model that allows certificate verification to be selectively
>> turned off for particular applications after upgrading to a version of
>> Python that verifies certificates by default
>> * the ability for all users to configure this setting on a per-application
>> basis, rather than on a per-system, or per-Python-installation basis
>>
>> This approach may be used for any redistributor provided version of
>> Python 2.7,
>> including those that advertise themselves as providing Python 2.7.9 or
>> later.
>>
>>
>> Required marker attribute
>> -------------------------
>>
>> The required marker attribute on the ``ssl`` module when implementing this
>> recommendation is::
>>
>> _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>
>> This not only makes it straightforward to detect the presence (or
>> absence) of
>> the capability, it also makes it possible to programmatically determine
>> the
>> relevant environment variable name.
>>
>>
>> Recommended modifications to the Python standard library
>> --------------------------------------------------------
>>
>> The recommended approach to providing a per-application configuration
>> setting
>> for HTTPS certificate verification that doesn't require modifications to
>> the
>> application itself is to:
>>
>> * modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
>> variable when the module is first imported into a Python process
>> * set the ``ssl._create_default_https_context`` function to be an alias
>> for
>> ``ssl._create_unverified_context`` if this environment variable is
>> present
>> and set to ``'0'``
>> * otherwise, set the ``ssl._create_default_https_context`` function to be
>> an
>> alias for ``ssl.create_default_context`` as usual
>>
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>> _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>>
>> def _get_https_context_factory():
>> config_setting = os.environ.get(_https_verify_envvar)
>> if config_setting == '0':
>> return _create_unverified_context
>> return create_default_context
>>
>> _create_default_https_context = _get_https_context_factory()
>>
>>
>> Security Considerations
>> -----------------------
>>
>> Relative to an unmodified version of CPython 2.7.9 or later, this approach
>> does introduce a new downgrade attack against the default security
>> settings
>> that potentially allows a sufficiently determined attacker to revert
>> Python
>> to the vulnerable configuration used in CPython 2.7.8 and earlier
>> releases.
>> However, such an attack requires the ability to modify the execution
>> environment of a Python process prior to the import of the ``ssl`` module,
>> and any attacker with such access would already be able to modify the
>> behaviour of the underlying OpenSSL implementation.
>>
>>
>> Recommendation for backporting to earlier Python versions
>> =========================================================
>>
>> Some redistributors, most notably Linux distributions, may choose to
>> backport
>> the PEP 476 HTTPS verification changes to modified Python versions based
>> on
>> earlier Python 2 maintenance releases. In these cases, a configuration
>> mechanism is needed that provides:
>>
>> * an opt-in model that allows the decision to enable HTTPS certificate
>> verification to be made independently of the decision to upgrade to the
>> Python version where the feature was first backported
>> * the ability for system administrators to set the default behaviour of
>> Python
>> applications and scripts run directly in the system Python installation
>> * the ability for the redistributor to consider changing the default
>> behaviour
>> of *new* installations at some point in the future without impacting
>> existing
>> installations that have been explicitly configured to skip verifying
>> HTTPS
>> certificates by default
>>
>> This approach should not be used for any Python installation that
>> advertises
>> itself as providing Python 2.7.9 or later, as most Python users will have
>> the
>> reasonable expectation that all such environments will validate HTTPS
>> certificates by default.
>>
>>
>> Required marker attribute
>> -------------------------
>>
>> The required marker attribute on the ``ssl`` module when implementing this
>> recommendation is::
>>
>> _cert_verification_config = '<path to configuration file>'
>>
>> This not only makes it straightforward to detect the presence (or
>> absence) of
>> the capability, it also makes it possible to programmatically determine
>> the
>> relevant configuration file name.
>>
>>
>> Recommended modifications to the Python standard library
>> --------------------------------------------------------
>>
>> The recommended approach to backporting the PEP 476 modifications to an
>> earlier
>> point release is to implement the following changes relative to the
>> default
>> PEP 476 behaviour implemented in Python 2.7.9+:
>>
>> * modify the ``ssl`` module to read a system wide configuration file when
>> the
>> module is first imported into a Python process
>> * define a platform default behaviour (either verifying or not verifying
>> HTTPS
>> certificates) to be used if this configuration file is not present
>> * support selection between the following three modes of operation:
>>
>> * ensure HTTPS certificate verification is enabled
>> * ensure HTTPS certificate verification is disabled
>> * delegate the decision to the redistributor providing this Python
>> version
>>
>> * set the ``ssl._create_default_https_context`` function to be an alias
>> for
>> either ``ssl.create_default_context`` or
>> ``ssl._create_unverified_context``
>> based on the given configuration setting.
>>
>>
>> Recommended file location
>> -------------------------
>>
>> This approach is currently only defined for \*nix system Python
>> installations.
>>
>> The recommended configuration file name is
>> ``/etc/python/cert-verification.cfg``.
>>
>> The ``.cfg`` filename extension is recommended for consistency with the
>> ``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
>>
>>
>> Recommended file format
>> -----------------------
>>
>> The configuration file should use a ConfigParser ini-style format with a
>> single section named ``[https]`` containing one required setting
>> ``verify``.
>>
>> Permitted values for ``verify`` are:
>>
>> * ``enable``: ensure HTTPS certificate verification is enabled by default
>> * ``disable``: ensure HTTPS certificate verification is disabled by
>> default
>> * ``platform_default``: delegate the decision to the redistributor
>> providing
>> this particular Python version
>>
>> If the ``[https]`` section or the ``verify`` setting are missing, or if
>> the
>> ``verify`` setting is set to an unknown value, it should be treated as if
>> the
>> configuration file is not present.
>>
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>> _cert_verification_config = '/etc/python/cert-verification.cfg'
>>
>> def _get_https_context_factory():
>> # Check for a system-wide override of the default behaviour
>> context_factories = {
>> 'enable': create_default_context,
>> 'disable': _create_unverified_context,
>> 'platform_default': _create_unverified_context, # For now :)
>> }
>> import ConfigParser
>> config = ConfigParser.RawConfigParser()
>> config.read(_cert_verification_config)
>> try:
>> verify_mode = config.get('https', 'verify')
>> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>> verify_mode = 'platform_default'
>> default_factory = context_factories.get('platform_default')
>> return context_factories.get(verify_mode, default_factory)
>>
>> _create_default_https_context = _get_https_context_factory()
>>
>>
>> Security Considerations
>> -----------------------
>>
>> The specific recommendations for the backporting case are designed to
>> work for
>> privileged, security sensitive processes, even those being run in the
>> following
>> locked down configuration:
>>
>> * run from a locked down administrator controlled directory rather than a
>> normal
>> user directory (preventing ``sys.path[0]`` based privilege escalation
>> attacks)
>> * run using the ``-E`` switch (preventing ``PYTHON*`` environment
>> variable based
>> privilege escalation attacks)
>> * run using the ``-s`` switch (preventing user site directory based
>> privilege
>> escalation attacks)
>> * run using the ``-S`` switch (preventing ``sitecustomize`` based
>> privilege
>> escalation attacks)
>>
>> The intent is that the *only* reason HTTPS verification should be getting
>> turned off system wide when using this approach is because:
>>
>> * an end user is running a redistributor provided version of CPython
>> rather
>> than running upstream CPython directly
>> * that redistributor has decided to provide a smoother migration path to
>> verifying HTTPS certificates by default than that being provided by the
>> upstream project
>> * either the redistributor or the local infrastructure administrator has
>> determined that it is appropriate to override the default upstream
>> behaviour
>> (at least for the time being)
>>
>> Using an administrator controlled configuration file rather than an
>> environment
>> variable has the essential feature of providing a smoother migration
>> path, even
>> for applications being run with the ``-E`` switch.
>>
>>
>> Combining the recommendations
>> =============================
>>
>> If a redistributor chooses to implement both recommendations, then the
>> environment variable should take precedence over the system-wide
>> configuration
>> setting. This allows the setting to be changed for a given user, virtual
>> environment or application, regardless of the system-wide default
>> behaviour.
>>
>> In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is
>> defined, and
>> set to anything *other* than ``'0'``, then HTTPS certificate verification
>> should be enabled.
>>
>> Example implementation
>> ----------------------
>>
>> ::
>>
>> _https_verify_envvar = 'PYTHONHTTPSVERIFY'
>> _cert_verification_config = '/etc/python/cert-verification.cfg'
>>
>> def _get_https_context_factory():
>> # Check for am environmental override of the default behaviour
>> config_setting = os.environ.get(_https_verify_envvar)
>> if config_setting is not None:
>> if config_setting == '0':
>> return _create_unverified_context
>> return create_default_context
>>
>> # Check for a system-wide override of the default behaviour
>> context_factories = {
>> 'enable': create_default_context,
>> 'disable': _create_unverified_context,
>> 'platform_default': _create_unverified_context, # For now :)
>> }
>> import ConfigParser
>> config = ConfigParser.RawConfigParser()
>> config.read(_cert_verification_config)
>> try:
>> verify_mode = config.get('https', 'verify')
>> except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
>> verify_mode = 'platform_default'
>> default_factory = context_factories.get('platform_default')
>> return context_factories.get(verify_mode, default_factory)
>>
>> _create_default_https_context = _get_https_context_factory()
>>
>>
>> Copyright
>> =========
>>
>> This document has been placed into the public domain.
>>
>>
>> --
>> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20151123/d9a50539/attachment.html>
More information about the Python-Dev
mailing list