[Python-Dev] PYTHONHTTPSVERIFY env var

[Nick Coghlan]

On 12 May 2015 at 04:49, M.-A. Lemburg <mal at egenix.com> wrote:
> On 11.05.2015 12:15, Nick Coghlan wrote:
>> By contrast, the configuration file shouldn't provide a new attack
>> vector (or simplify any existing attack vector), as if you have the
>> permissions needed to modify the config file, you likely also have the
>> permissions needed to modify the system certificate store, and the
>> latter is a *far* more interesting attack vector than a downgrade
>> attack solely against Python.
>>
>> Thus the environment variable based off switch is neither necessary
>> (as an administrator controlled configuration file can do the job),
>> nor sufficient (it can't handle the -E switch), *and* it represents an
>> increase in the attack surface area relative to a Python
>> implementation without the capability.
>
> Whether or not -E will have an effect on the env var depends
> on the implementation. At the moment, -E only has an effect
> on the C runtime, while the stdlib happily reads from os.environ
> without taking the flag into account.

I had an off-list discussion with Christian Heimes about that in
relation to the OpenSSL flags, and he pointed out the reason -E
specifically needs to be a command line switch is because that is the
only way to affect how environment variables are processed during
interpreter startup. Once an application is up and running, further
environment variable sanitisation can be handled at an application
level by whitelisting entries in os.environ and deleting everything
else.

> As proposed, the PYTHONHTTPSVERIFY would only affect the ssl
> module and only be checked when loading this module, i.e. not
> at Python startup time.

Right, the same is true for the configuration file proposal.

>> It would be nice to hear from ActiveState, Enthought & Continuum
>> Analytics as well, but if they don't chime in here, I don't see any
>> particular need to go chasing them explicitly.
>
> I think the approach to only consider a subset of redistributors
> as viable targets for such a switch is a bit too narrow.
>
> You are leaving out all the parties which use custom
> Python installations to run their applications, e.g.
> the Plone and Zope community, the ZenOSS community,
> the many Windows applications built on Python, etc.

No, they already have a solution: monkeypatch (or just plain patch)
the SSL module. That's an upstream supported technique, which is why
it's documented in the PEP.

The problem we (as in Red Hat) ran into was that that technique
*doesn't work* for the case of backporting PEP 476 to Python 2.7.5 as
an opt-in feature.

>>> to fix this regression in 2.7.9.
>>
>> We made the decision when PEP 476 was accepted that this change turned
>> a silent security failure into a noisy one, rather than being a
>> regression in its own right. PEP 493 isn't about disagreeing with that
>> decision, it's about providing a smoother upgrade path in contexts
>> where letting the security failure remain silent is deemed to be
>> preferred in the near term.
>
> The change wasn't regression. The missing downgrade path
> is a regression.

It's a shame we don't have "-X" options in Python 2, as that would be
a nice hard-to-attack option (although it wouldn't play well with
subprocesses)

> Some other comments on PEP 493:
>
> * I don't think we really want to add the overhead of
>   having to parse an INI file every time Python starts up.
>   Please remember that we just parsing of the sysconfig
>   data not long ago because we wanted to avoid this startup
>   time.

Compared to the overhead of reading from the system cert database,
reading a config file at ssl module import time should be trivial.

> * I don't see why the attack surface of using an INI file
>   somewhere in the system should be smaller than e.g. using
>   sitecustomize.py

You can put sitecustomize.py in a user directory, and if there's no
system wide sitecustomize, Python will read it automatically (unless
user site directories are turned off).

> * If done right, we'd also need a switch to ignore this
>   global config file and recommend using it to reduce the
>   attack surface (for the same reason you explain in the
>   PEP)

No, the recommendation there would be to upgrade to a newer version of
Python that doesn't offer this downgrade capability. It's a proposal
for a transition smoothing technique, not a permanent capability (I
did suggest the latter at one point, but the discussion on the issue
tracker persuaded me that was a bad idea, with the increased attack
surface being a key part of that change of heart).

> * I don't think a global switch is the right way forward.
>   Many applications on properly configured systems will
>   work fine with the new default. The downgrade option is
>   only needed for those cases, where they don't and you
>   don't have a good way to fix the application.

And techniques like chroots and containers let you do that selectively.

The key thing I'm after is an agreed technique for backporting to
earlier 2.7.x releases that allows PEP 476 to be provided as an opt-in
capability, rather than gating it on folks upgrading to 2.7.9, which
isn't going to happen for *years* in a great many environments (Ubuntu
14.04 LTS, for example, doesn't go end of life until 2019 and ships
2.7.6 + non-intrusive security patches, while RHEL 7 doesn't go end of
life until 2024 and ships 2.7.5 + compatible backports that address
customer problems).

While other Linux vendors have currently decided to leave the HTTPS
problem unfixed in their long term support releases (due to the risk
of causing service failures in customer environments), I'm hoping they
may revisit those decisions if there's a specific technique already
agreed with upstream for backporting the capability in a way that
makes it an opt-in feature that customers can switch on independently
of the inclusion of the feature backport in the system Python.

> * Most applications use some kind of virtualenv
>   Python environment to run the code. These are
>   typically isolated from the system Python installation
>   and so wouldn't want to use a system wide global INI
>   neither.

Having a separate configuration setting that controls verification in
virtual environments is a good idea. That could also provide the
per-application opt-out capability that you're after.

> * The -S switch completely disables importing site.py.
>   That's not really a viable solution in the age of
>   pip - your local installation wouldn't find the installed
>   packages anymore, since these are installed in site-packages/
>   which again, is set up by site.py.

Yes, getting an administrative application to the point where -S can
be used means getting it to a point where it has *no* Python
dependencies outside the standard library. It can certainly be done,
but often won't be worth the hassle. As a result, using -s to turn off
the user site directory and -E to turn off PYTHONPATH processing are
the more common sys.path related hardening techniques in Python 2.7.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia