From cory at lukasa.co.uk Wed Jan 11 14:01:14 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 11 Jan 2017 19:01:14 +0000 Subject: [Security-sig] Unified TLS API for Python Message-ID: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> All, A recent change in the Fastly policy has led the distutils-sig to discover that there is a very real risk of many macOS installations of Python being entirely unable to access PyPI using pip, due to the fact that they are linked against the ancient system OpenSSL (see [1]). This problem ultimately stems from the fact that Python has never exposed the ability to use TLS backends other than OpenSSL in the standard library. As Christian and I discussed back at PyCon US 2015, the ssl module would more properly be called the openssl module, due to exposing many OpenSSL-specific concepts and behaviours in its API. This has meant that the Python ecosystem is overwhelmingly also an OpenSSL ecosystem, which is problematic on Windows and macOS (and Unices that for whatever reason aren?t interested in shipping an OpenSSL), as it has meant that Python needs to bring its own OpenSSL, and that it is troublesome to interact with the system trust database. The first step to resolving this would be to provide a new module that exposes TLS concepts in a much more generic manner. There are two possible approaches to this. The first is to have this module be a generic concrete implementation that can be compiled against multiple TLS backends (like curl). This would require that all the relevant bindings be built into the standard library from the get-go, which provides a substantial maintenance burden on a team of people that are already understaffed maintaining the ssl module alone. The second approach is to define a generic high-level TLS interface that provides a minimal usable interface that can be implemented by both first- and third-party modules. This would allow the standard library to continue to ship with only exactly what it needs (for example, OpenSSL, SecureTransport and SChannel), but would give those who want to use more esoteric TLS choices (NSS, GnuTLS, mbedTLS are some examples) an API that they can implement that will guarantee that complying modules can use the appropriate TLS backend. To that end, I?ve composed a draft PEP that would define this API. The current copy can be found on GitHub[2]. I?d like to see if there is interest from the Python security community in pursuing this approach to generic TLS, and what changes would be necessary to this API before we submitted it to python-dev. Please let me know what you think. Cory [1]: https://mail.python.org/pipermail/distutils-sig/2017-January/029970.html [2]: https://github.com/Lukasa/peps/pull/1 - - - Abstract ======== This PEP would define a standard TLS interface in the form of a collection of abstract base classes. This interface would allow Python implementations and third-party libraries to provide bindings to TLS libraries other than OpenSSL that can be used by tools that expect the interface provided by the Python standard library, with the goal of reducing the dependence of the Python ecosystem on OpenSSL. Rationale ========= In the 21st century it has become increasingly clear that robust and user-friendly TLS support is an extremely important part of the ecosystem of any popular programming language. For most of its lifetime, this role in the Python ecosystem has primarily been served by the `ssl module`_, which provides a Python API to the `OpenSSL library`_. Because the ``ssl`` module is distributed with the Python standard library, it has become the overwhelmingly most-popular method for handling TLS in Python. An extraordinary majority of Python libraries, both in the standard library and on the Python Package Index, rely on the ``ssl`` module for their TLS connectivity. Unfortunately, the preeminence of the ``ssl`` module has had a number of unforseen side-effects that have had the effect of tying the entire Python ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even in situations where it may provide a worse user experience than alternative TLS implementations, which imposes a cognitive burden and makes it hard to provide "platform-native" experiences. Problems -------- The fact that the ``ssl`` module is build into the standard library has meant that all standard-library Python networking libraries are entirely reliant on the OpenSSL that the Python implementation has been linked against. This leads to the following issues: * It is difficult to take advantage of new, higher-security TLS without recompiling Python to get a new OpenSSL. While there are third-party bindings to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format that the standard library understands, forcing projects that want to use them to maintain substantial compatibility layers. * For Windows distributions of Python, they need to be shipped with a copy of OpenSSL. This puts the CPython development team in the position of being OpenSSL redistributors, potentially needing to ship security updates to the Windows Python distributions when OpenSSL vulnerabilities are released. * For macOS distributions of Python, they need either to be shipped with a copy of OpenSSL or linked against the system OpenSSL library. Apple has formally deprecated linking against the system OpenSSL library, and even if they had not, that library version has been unsupported by upstream for nearly one year as of the time of writing. The CPython development team has started shipping newer OpenSSLs with the Python available from python.org, but this has the same problem as with Windows. * Many systems, including but not limited to Windows and macOS, do not make their system certificate stores available to OpenSSL. This forces users to either obtain their trust roots from elsewhere (e.g. `certifi`_) or to attempt to export their system trust stores in some form. Relying on `certifi`_ is less than ideal, as most system administrators do not expect to receive security-critical software updates from PyPI. Additionally, it is not easy to extend the `certifi`_ trust bundle to include custom roots, or to centrally manage trust using the `certifi`_ model. Even in situations where the system certificate stores are made available to OpenSSL in some form, the experience is still sub-standard, as OpenSSL will perform different validation checks than the platform-native TLS implementation. This can lead to users experiencing different behaviour on their browsers or other platform-native tools than they experience in Python, with little or no recourse to resolve the problem. * Users may wish to integrate with TLS libraries other than OpenSSL for many other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), or because OpenSSL is simply too large and unweildy for the platform (e.g. for embedded Python). Those users are left with the requirement to use third-party networking libraries that can interact with their preferred TLS library or to shim their preferred library into the OpenSSL-specific ``ssl`` module API. Additionally, the ``ssl`` module as implemented today limits the ability of CPython itself to add support for alternative TLS backends, or remove OpenSSL support entirely, should either of these become necessary or useful. The ``ssl`` module exposes too many OpenSSL-specific function calls and features to easily map to an alternative TLS backend. Proposal ======== This PEP proposes to introduce a few new Abstract Base Classes in Python 3.7 to provide TLS functionality that is not so strongly tied to OpenSSL. It also proposes to update standard library modules to use only the interface exposed by these abstract base classes wherever possible. There are three goals here: 1. To provide a common API surface for both core and third-party developers to target their TLS implementations to. This allows TLS developers to provide interfaces that can be used by most Python code, and allows network developers to have an interface that they can target that will work with a wide range of TLS implementations. 2. To provide an API that has few or no OpenSSL-specific concepts leak through. The ``ssl`` module today has a number of warts caused by leaking OpenSSL concepts through to the API: the new ABCs would remove those specific concepts. 3. To provide a path for the core development team to make OpenSSL one of many possible TLS backends, rather than requiring that it be present on a system in order for Python to have TLS support. The proposed interface is laid out below. Abstract Base Classes --------------------- There are several interfaces that require standardisation. Those interfaces are: 1. Configuring TLS, currently implemented by the `SSLContext`_ class in the ``ssl`` module. 2. Wrapping a socket object, currently implemented by the `SSLSocket`_ class in the ``ssl`` module. 3. Providing an in-memory buffer for doing in-memory encryption or decryption with no actual I/O (necessary for asynchronous I/O models), currently implemented by the `SSLObject`_ class in the ``ssl`` module. 4. Specifying TLS cipher suites. There is currently no code for doing this in the standard library: instead, the standard library uses OpenSSL cipher suite strings. 5. Specifying application-layer protocols that can be negotiated during the TLS handshake. 6. Specifying TLS versions. 7. Reporting errors to the caller, currently implemented by the `SSLError`_ class in the ``ssl`` module. While it is technically possible to define (2) in terms of (3), for the sake of simplicity it is easier to define these as two separate ABCs. Implementations are of course free to implement the concrete subclasses however they see fit. Obviously, (4) doesn't require an abstract base class: instead, it requires a richer API for configuring supported cipher suites that can be easily updated with supported cipher suites for different implementations. Context ~~~~~~~ The ``Context`` abstract base class defines an object that allows configuration of TLS. It can be thought of as a factory for ``TLSWrappedSocket`` and ``TLSWrappedBuffer`` objects. The ``Context`` abstract base class has the following class definition:: TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] ServerNameCallback = Callable[[TLSBufferObject, Optional[str], Context], Any] class _BaseContext(metaclass=ABCMeta): @property @abstractmethod def validate_certificates(self) -> bool: """ Whether to validate the TLS certificates. This switch operates at a very broad scope: either validation is enabled, in which case all forms of validation are performed including hostname validation if possible, or validation is disabled, in which case no validation is performed. Not all backends support having their certificate validation disabled. If a backend does not support having their certificate validation disabled, attempting to set this property to ``False`` will throw a ``TLSError``. """ @validate_certificates.setter @abstractmethod def validate_certificates(self, val: bool) -> None: pass @abstractmethod def register_certificates(self, certificates: str, key=None: Optional[str], password=None: Optional[Callable[[], Union[AnyStr, bytearray]]]) -> None: """ Loads a certificate, a number of intermediate certificates, and the corresponding private key. These certificates will be offered to the remote peer during the handshake if required. The ``certificates`` argument must be a bytestring containing the PEM-encoded certificates. The first PEM-encoded certificate must be the leaf certificate. All subsequence certificates will be offered as intermediate additional certificates. The ``key`` argument, if present, must contain the PEM-encoded private key associated with the leaf certificate. If not present, the private key will be extracted from ``certificates``. The ``password`` argument may be a function to call to get the password for decrypting the private key. It will only be called if the private key is encrypted and a password is necessary. It will be called with no arguments, and it should return a string, bytes, or bytearray. If the return value is a string it will be encoded as UTF-8 before using it to decrypt the key. Alternatively a string, bytes, or bytearray value may be supplied directly as the password argument. It will be ignored if the private key is not encrypted and no password is needed. """ @abstractmethod def set_ciphers(self, ciphers: List[Ciphers]) -> None: """ Set the available ciphers for TLS connections created with this context. ``ciphers`` should be a list of ciphers from the ``Cipher`` registry. If none of the ``ciphers`` provided to this object are supported or available, a ``TLSError`` will be raised. """ @abstractmethod def set_inner_protocols(self, protocols: List[NextProtocol]) -> None: """ Specify which protocols the socket should advertise as supported during the TLS handshake. This may be advertised using either or both of ALPN or NPN. ``protocols`` should be a list of acceptable protocols in the form of ``NextProtocol`` objects, such as ``[H2, HTTP1]``, ordered by preference. The selection of the protocol will happen during the handshake, and will use whatever protocol negotiation mechanisms are available and supported by both peers. If the TLS implementation doesn't support protocol negotiation, this method will raise ``NotImplementedError``. """ @abstractmethod def set_sni_callback(self, callback: Optional[ServerNameCallback]) -> None: """ Register a callback function that will be called after the TLS Client Hello handshake message has been received by the TLS server when the TLS client specifies a server name indication. Only one callback can be set per ``Context``. If ``callback`` is ``None`` then the callback is disabled. Calling this function a subsequent time will disable the previously registered callback. The ``callback`` function will be called with three arguments: the first will be the ``TLSBufferObject`` for the connection; the second will be a string that represents the server name that the client is intending to communicate (or ``None`` if the TLS Client Hello does not contain a server name); and the third argument will be the original ``Context``. The server name argument will be the IDNA *decoded* server name. The ``callback`` must return ``None`` to allow negotiation to continue. Other return values signal errors. Attempting to control what error is signaled by the underlying TLS implementation is not specified in this API, but is up to the concrete implementation to handle. """ @abstractmethod def set_version_range(self, lower_bound=None: Optional[TLSVersion], upper_bound=None: Optional[TLSVersion]) -> None: """ Set the minumum and maximum versions of TLS that should be allowed on TLS connections made by this context. If present, ``lower_bound`` will set the lowest acceptable TLS version. If present, ``upper_bound`` will set the highest acceptable TLS version. If either argument is ``None``, this will leave that bound unchanged. """ @abstractmethod def wrap_socket(self, socket: socket.socket, server_side=False: bool, auto_handshake=True: bool, server_hostname=None: Optional[str]) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``server_side`` is a boolean which identifies whether server-side or client-side behavior is desired from this socket. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.connect()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. On client connections, the optional parameter ``server_hostname`` specifies the hostname of the service which we are connecting to. This allows a single server to host multiple SSL-based services with distinct certificates, quite similarly to HTTP virtual hosts. Specifying ``server_hostname`` will raise a ValueError if ``server_side`` is ``True``. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any, server_side=False: bool, server_hostname=None: Optional[str]) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. The ``server_side`` and ``server_hostname`` parameters have the same meaning as in ``wrap_socket``. """ class ClientContext(metaclass=ABCMeta): @abstractmethod def wrap_socket(self, socket: socket.socket, auto_handshake=True: bool, server_hostname=None: Optional[str]) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.connect()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. The optional parameter ``server_hostname`` specifies the hostname of the service which we are connecting to. This allows a single server to host multiple SSL-based services with distinct certificates, quite similarly to HTTP virtual hosts. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any, server_hostname=None: Optional[str]) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. The ``server_hostname`` parameter has the same meaning as in ``wrap_socket``. """ class ServerContext(metaclass=ABCMeta): @abstractmethod def wrap_socket(self, socket: socket.socket, auto_handshake=True: bool) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.connect()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. """ Socket ~~~~~~ The socket-wrapper ABC will be defined by the ``TLSWrappedSocket`` ABC, which has the following definition:: class TLSWrappedSocket(metaclass=ABCMeta): # The various socket methods all must be implemented. Their definitions # have been elided from this class defintion in the PEP because they # aren't instructive. @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[Cipher]: """ Returns the Cipher entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[NextProtocol]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @context.setter @abstractmethod def context(self, value: Context) -> None: """ Set the value of the ``Context`` object this socket is tied to. This operation (changing the context) may not always be supported. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ Buffer ~~~~~~ The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, which has the following definition:: class TLSWrappedBuffer(metaclass=ABCMeta): @abstractmethod def read(self, len=1024: int, buffer=None: Any) -> Union[bytes, int]: """ Read up to ``len`` bytes of data from the input buffer and return the result as a ``bytes`` instance. If ``buffer`` is specified, then read into the buffer instead, and return the number of bytes read. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``read()`` can also cause write operations. """ @abstractmethod def write(self, buf: Any) -> int: """ Write ``buf`` in encrypted form to the output buffer and return the number of bytes written. The ``buf`` argument must be an object supporting the buffer interface. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``write()`` can also cause read operations. """ @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[Cipher]: """ Returns the Cipher entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[NextProtocol]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @context.setter @abstractmethod def context(self, value: Context) -> None: """ Set the value of the ``Context`` object this socket is tied to. This operation (changing the context) may not always be supported. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ Cipher Suites ~~~~~~~~~~~~~ Todo Protocol Negotiation ~~~~~~~~~~~~~~~~~~~~ Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 handshake. While NPN and ALPN are, at their fundamental level, built on top of bytestrings, string-based APIs are frequently problematic as they allow for errors in typing that can be hard to detect. For this reason, this module would define a type that protocol negotiation implementations can pass and be passed. This type would wrap a bytestring, but allow for aliases for well-known protocols. This allows us to avoid the problems inherent in typos for well-known protocols, while allowing the full extensibility of the protocol negotiation layer if needed. :: NextProtocol = namedtuple('NextProtocol', ['name']) H2 = NextProtocol(b'h2') H2C = NextProtocol(b'h2c') HTTP1 = NextProtocol(b'http/1.1') WEBRTC = NextProtocol(b'webrtc') C_WEBRTC = NextProtocol(b'c-webrtc') FTP = NextProtocol(b'ftp') STUN = NextProtocol(b'stun.nat-discovery') TURN = NextProtocol(b'stun.turn') TLS Versions ~~~~~~~~~~~~ It is often useful to be able to restrict the versions of TLS you're willing to support. There are many security advantages in refusing to use old versions of TLS, and some misbehaving servers will mishandle TLS clients advertising support for newer versions. The following enumerated type can be used to gate TLS versions. Forward-looking applications should almost never set a maximum TLS version unless they absolutely must, as a TLS backend that is newer than the Python that uses it may support TLS versions that are not in this enumerated type. Additionally, this enumerated type defines two additional flags that can always be used to request either the lowest or highest TLS version supported by an implementation. :: class TLSVersion(Enum): MINIMUM_SUPPORTED SSLv2 SSLv3 TLSv1 TLSv1_1 TLSv1_2 TLSv1_3 MAXIMUM_SUPPORTED Errors ~~~~~~ This module would define three base classes for use with error handling. Unlike the other classes defined here, these classes are not *abstract*, as they have no behaviour. They exist simply to signal certain common behaviours. Backends should subclass these exceptions in their own packages, but needn't define any behaviour for them. These exceptions should *never* be thrown directly, they should always be subclassed. The definitions of the errors are below:: class TLSError(Exception): """ The base exception for all TLS related errors from any backend. Catching this error should be sufficient to catch *all* TLS errors, regardless of what backend is used. """ class WantWriteError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is written to the network, or until the output buffer is drained. """ class WantReadError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is read from the network, or until more data is available in the input buffer. """ Changes to the Standard Library =============================== The portions of the standard library that interact with TLS should be revised to use these ABCs. This will allow them to function with other TLS backends. This includes the following modules: - asyncio - ftplib - http.client - imaplib - nntplib - poplib - smtplib Future ====== Major future TLS features may require revisions of these ABCs. These revisions should be made cautiously: many backends may not be able to move forward swiftly, and will be invalidated by changes in these ABCs. This is acceptable, but wherever possible features that are specific to individual implementations should not be added to the ABCs. The ABCs should restrict themselves to high-level descriptions of IETF-specified features. ToDo ==== * Consider adding a new parameter (``valid_subjects``?) to ``wrap_socket`` and ``wrap_buffers`` that specifies in a *typed* manner what kind of entries in the SAN field are acceptable. This would break the union between SNI and cert validation, which may be a good thing (you can't SNI an IP address, but you can validate a cert with one if you want). * It's annoying that there's no type signature for fileobj. Do I really have to define one as part of this PEP? Otherwise, how do I define the types of the arguments to ``wrap_buffers``? * Do we need ways to control hostname validation? * Do we need to support getpeercert? Should we always return DER instead of the weird semi-structured thing? * How do we load certs from locations on disk? What about HSMs? * How do we signal to load certs from the OS? What happens if an implementation doesn't let you *not* load those certs? References ========== .. _ssl module: https://docs.python.org/3/library/ssl.html .. _OpenSSL Library: https://www.openssl.org/ .. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ .. _certifi: https://pypi.org/project/certifi/ .. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext .. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket .. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject .. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError Copyright ========= This document has been placed in the public domain. From ethan at stoneleaf.us Wed Jan 11 14:36:00 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Jan 2017 11:36:00 -0800 Subject: [Security-sig] new TLS module Message-ID: <58768920.6040701@stoneleaf.us> With TLSv1.2 becoming mandatory in the near future and the various issues with the ssl module there is more interest in a tls module for Python. What should such a module include? What should it not include? Any reason to not also have a backport of this module? -- ~Ethan~ From christian at cheimes.de Wed Jan 11 14:36:10 2017 From: christian at cheimes.de (Christian Heimes) Date: Wed, 11 Jan 2017 20:36:10 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> Message-ID: <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> On 2017-01-11 20:01, Cory Benfield wrote: > The ``Context`` abstract base class defines an object that allows configuration > of TLS. It can be thought of as a factory for ``TLSWrappedSocket`` and > ``TLSWrappedBuffer`` objects. > > The ``Context`` abstract base class has the following class definition:: > > TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] > ServerNameCallback = Callable[[TLSBufferObject, Optional[str], Context], Any] > > class _BaseContext(metaclass=ABCMeta): > > @property > @abstractmethod > def validate_certificates(self) -> bool: > """ > Whether to validate the TLS certificates. This switch operates at a > very broad scope: either validation is enabled, in which case all > forms of validation are performed including hostname validation if > possible, or validation is disabled, in which case no validation is > performed. > > Not all backends support having their certificate validation > disabled. If a backend does not support having their certificate > validation disabled, attempting to set this property to ``False`` > will throw a ``TLSError``. > """ > > @validate_certificates.setter > @abstractmethod > def validate_certificates(self, val: bool) -> None: > pass For 3.7 I'm planning to replace ssl.match_hostname() with OpenSSL 1.0.2's API. For now the one flag is enough. Later we can discuss settings for wildcard, IP address and CN matching. > > @abstractmethod > def register_certificates(self, > certificates: str, > key=None: Optional[str], > password=None: Optional[Callable[[], Union[AnyStr, bytearray]]]) -> None: > """ > Loads a certificate, a number of intermediate certificates, and the > corresponding private key. These certificates will be offered to > the remote peer during the handshake if required. > > The ``certificates`` argument must be a bytestring containing the > PEM-encoded certificates. The first PEM-encoded certificate must be > the leaf certificate. All subsequence certificates will be offered > as intermediate additional certificates. > > The ``key`` argument, if present, must contain the PEM-encoded > private key associated with the leaf certificate. If not present, > the private key will be extracted from ``certificates``. > > The ``password`` argument may be a function to call to get the > password for decrypting the private key. It will only be called if > the private key is encrypted and a password is necessary. It will > be called with no arguments, and it should return a string, bytes, > or bytearray. If the return value is a string it will be encoded as > UTF-8 before using it to decrypt the key. Alternatively a string, > bytes, or bytearray value may be supplied directly as the password > argument. It will be ignored if the private key is not encrypted > and no password is needed. > """ I don't think this function works for all libraries and use cases. For some implementations the order of certificates is very important. For NSS and PKCS#11 we rather need to specify the slot or nick name of the cert. For 3.7 I also like to introduce X509 objects and EVP_Key wrapper, so this function would need to consume a stack of certificates. Since this function is only required for TLS servers and TLS client cert authentication, I'd rather mark this function provisional or not define it in the first version. > @abstractmethod > def set_ciphers(self, ciphers: List[Ciphers]) -> None: > """ > Set the available ciphers for TLS connections created with this > context. ``ciphers`` should be a list of ciphers from the > ``Cipher`` registry. If none of the ``ciphers`` provided to this > object are supported or available, a ``TLSError`` will be raised. > """ Implementors should initial context with sensible default settings, preferable system-wide settings. For example Fedora is currently implementing https://fedoraproject.org/wiki/Changes/CryptoPolicy for OpenSSL, NSS and GnuTLS. > > @abstractmethod > def set_inner_protocols(self, protocols: List[NextProtocol]) -> None: > """ > Specify which protocols the socket should advertise as supported > during the TLS handshake. This may be advertised using either or > both of ALPN or NPN. > > ``protocols`` should be a list of acceptable protocols in the form > of ``NextProtocol`` objects, such as ``[H2, HTTP1]``, ordered by > preference. The selection of the protocol will happen during the > handshake, and will use whatever protocol negotiation mechanisms > are available and supported by both peers. > > If the TLS implementation doesn't support protocol negotiation, > this method will raise ``NotImplementedError``. > """ > > @abstractmethod > def set_sni_callback(self, callback: Optional[ServerNameCallback]) -> None: > """ > Register a callback function that will be called after the TLS > Client Hello handshake message has been received by the TLS server > when the TLS client specifies a server name indication. > > Only one callback can be set per ``Context``. If ``callback`` is > ``None`` then the callback is disabled. Calling this function a > subsequent time will disable the previously registered callback. > > The ``callback`` function will be called with three arguments: the > first will be the ``TLSBufferObject`` for the connection; the > second will be a string that represents the server name that the > client is intending to communicate (or ``None`` if the TLS Client > Hello does not contain a server name); and the third argument will > be the original ``Context``. The server name argument will be the > IDNA *decoded* server name. > > The ``callback`` must return ``None`` to allow negotiation to > continue. Other return values signal errors. Attempting to control > what error is signaled by the underlying TLS implementation is not > specified in this API, but is up to the concrete implementation to > handle. > """ > > @abstractmethod > def set_version_range(self, lower_bound=None: Optional[TLSVersion], > upper_bound=None: Optional[TLSVersion]) -> None: > """ > Set the minumum and maximum versions of TLS that should be allowed > on TLS connections made by this context. > > If present, ``lower_bound`` will set the lowest acceptable TLS > version. If present, ``upper_bound`` will set the highest > acceptable TLS version. If either argument is ``None``, this will > leave that bound unchanged. > """ https://bugs.python.org/issue27876 > @abstractmethod > def wrap_socket(self, socket: socket.socket, server_side=False: bool, > auto_handshake=True: bool, > server_hostname=None: Optional[str]) -> TLSWrappedSocket: > """ > Wrap an existing Python socket object ``socket`` and return a > ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` > socket: all other socket types are unsupported. > > The returned SSL socket is tied to the context, its settings and > certificates. > > The parameter ``server_side`` is a boolean which identifies whether > server-side or client-side behavior is desired from this socket. > > The parameter ``auto_handshake`` specifies whether to do the SSL > handshake automatically after doing a ``socket.connect()``, or > whether the application program will call it explicitly, by > invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling > ``TLSWrappedSocket.do_handshake()`` explicitly gives the program > control over the blocking behavior of the socket I/O involved in > the handshake. > > On client connections, the optional parameter ``server_hostname`` > specifies the hostname of the service which we are connecting to. > This allows a single server to host multiple SSL-based services > with distinct certificates, quite similarly to HTTP virtual hosts. > Specifying ``server_hostname`` will raise a ValueError if > ``server_side`` is ``True``. > """ > > @abstractmethod > def wrap_buffers(self, incoming: Any, outgoing: Any, > server_side=False: bool, > server_hostname=None: Optional[str]) -> TLSWrappedBuffer: > """ > Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to > create an in-memory stream for TLS. The SSL routines will read data > from ``incoming`` and decrypt it, and write encrypted data to > ``outgoing``. > > The ``server_side`` and ``server_hostname`` parameters have the > same meaning as in ``wrap_socket``. > """ How about not defining this methods at all? IMO it makes no sense to support client and server connections from the same context. This is also the gist of a PEP I'm currently working on. Basically I want to get rid of all context protocols except for two: PROTOCOL_TLS_CLIENT and PROTOCOL_TLS_SERVER. The client protocol only supports the client side part of a TLS handshake and has sensible default settings for clients (require cert and hostname verification). The server protocol only supports the server side part of a TLS handshake and has cert validation disabled by default. This removes the need for server_side argument because the value can be inferred from the context. > class ClientContext(metaclass=ABCMeta): > @abstractmethod > def wrap_socket(self, socket: socket.socket, > auto_handshake=True: bool, > server_hostname=None: Optional[str]) -> TLSWrappedSocket: > """ > Wrap an existing Python socket object ``socket`` and return a > ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` > socket: all other socket types are unsupported. > > The returned SSL socket is tied to the context, its settings and > certificates. > > The parameter ``auto_handshake`` specifies whether to do the SSL > handshake automatically after doing a ``socket.connect()``, or > whether the application program will call it explicitly, by > invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling > ``TLSWrappedSocket.do_handshake()`` explicitly gives the program > control over the blocking behavior of the socket I/O involved in > the handshake. > > The optional parameter ``server_hostname`` specifies the hostname > of the service which we are connecting to. This allows a single > server to host multiple SSL-based services with distinct > certificates, quite similarly to HTTP virtual hosts. > """ > > @abstractmethod > def wrap_buffers(self, incoming: Any, outgoing: Any, > server_hostname=None: Optional[str]) -> TLSWrappedBuffer: > """ > Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to > create an in-memory stream for TLS. The SSL routines will read data > from ``incoming`` and decrypt it, and write encrypted data to > ``outgoing``. > > The ``server_hostname`` parameter has the same meaning as in > ``wrap_socket``. > """ > > > class ServerContext(metaclass=ABCMeta): > @abstractmethod > def wrap_socket(self, socket: socket.socket, > auto_handshake=True: bool) -> TLSWrappedSocket: > """ > Wrap an existing Python socket object ``socket`` and return a > ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` > socket: all other socket types are unsupported. > > The returned SSL socket is tied to the context, its settings and > certificates. > > The parameter ``auto_handshake`` specifies whether to do the SSL > handshake automatically after doing a ``socket.connect()``, or > whether the application program will call it explicitly, by > invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling > ``TLSWrappedSocket.do_handshake()`` explicitly gives the program > control over the blocking behavior of the socket I/O involved in > the handshake. > """ > > @abstractmethod > def wrap_buffers(self, incoming: Any, outgoing: Any) -> TLSWrappedBuffer: > """ > Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to > create an in-memory stream for TLS. The SSL routines will read data > from ``incoming`` and decrypt it, and write encrypted data to > ``outgoing``. > """ > > > Socket > ~~~~~~ > > The socket-wrapper ABC will be defined by the ``TLSWrappedSocket`` ABC, which > has the following definition:: > > class TLSWrappedSocket(metaclass=ABCMeta): > # The various socket methods all must be implemented. Their definitions > # have been elided from this class defintion in the PEP because they > # aren't instructive. > @abstractmethod > def do_handshake(self) -> None: > """ > Performs the TLS handshake. Also performs certificate validation > and hostname verification. > """ > > @abstractmethod > def cipher(self) -> Optional[Cipher]: > """ > Returns the Cipher entry for the cipher that has been negotiated on > the connection. If no connection has been negotiated, returns > ``None``. > """ > > @abstractmethod > def negotiated_protocol(self) -> Optional[NextProtocol]: > """ > Returns the protocol that was selected during the TLS handshake. > This selection may have been made using ALPN, NPN, or some future > negotiation mechanism. > > If ``Context.set_inner_protocols()`` was not called, if the other > party does not support protocol negotiation, if this socket does > not support any of the peer's proposed protocols, or if the > handshake has not happened yet, ``None`` is returned. > """ > > @property > @abstractmethod > def context(self) -> Context: > """ > The ``Context`` object this socket is tied to. > """ > > @context.setter > @abstractmethod > def context(self, value: Context) -> None: > """ > Set the value of the ``Context`` object this socket is tied to. > This operation (changing the context) may not always be supported. > """ > > @abstractproperty > def negotiated_tls_version(self) -> Optional[TLSVersion]: > """ > The version of TLS that has been negotiated on this connection. > """ > > > Buffer > ~~~~~~ > > The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, which > has the following definition:: > > class TLSWrappedBuffer(metaclass=ABCMeta): > @abstractmethod > def read(self, len=1024: int, buffer=None: Any) -> Union[bytes, int]: > """ > Read up to ``len`` bytes of data from the input buffer and return > the result as a ``bytes`` instance. If ``buffer`` is specified, > then read into the buffer instead, and return the number of bytes > read. > > Raise ``WantReadError`` or ``WantWriteError`` if there is > insufficient data in either the input or output buffer and the > operation would have caused data to be written or read. > > As at any time a re-negotiation is possible, a call to ``read()`` > can also cause write operations. > """ > > @abstractmethod > def write(self, buf: Any) -> int: > """ > Write ``buf`` in encrypted form to the output buffer and return the > number of bytes written. The ``buf`` argument must be an object > supporting the buffer interface. > > Raise ``WantReadError`` or ``WantWriteError`` if there is > insufficient data in either the input or output buffer and the > operation would have caused data to be written or read. > > As at any time a re-negotiation is possible, a call to ``write()`` > can also cause read operations. > """ > > @abstractmethod > def do_handshake(self) -> None: > """ > Performs the TLS handshake. Also performs certificate validation > and hostname verification. > """ > > @abstractmethod > def cipher(self) -> Optional[Cipher]: > """ > Returns the Cipher entry for the cipher that has been negotiated on > the connection. If no connection has been negotiated, returns > ``None``. > """ > > @abstractmethod > def negotiated_protocol(self) -> Optional[NextProtocol]: > """ > Returns the protocol that was selected during the TLS handshake. > This selection may have been made using ALPN, NPN, or some future > negotiation mechanism. > > If ``Context.set_inner_protocols()`` was not called, if the other > party does not support protocol negotiation, if this socket does > not support any of the peer's proposed protocols, or if the > handshake has not happened yet, ``None`` is returned. > """ > > @property > @abstractmethod > def context(self) -> Context: > """ > The ``Context`` object this socket is tied to. > """ > > @context.setter > @abstractmethod > def context(self, value: Context) -> None: > """ > Set the value of the ``Context`` object this socket is tied to. > This operation (changing the context) may not always be supported. > """ > > @abstractproperty > def negotiated_tls_version(self) -> Optional[TLSVersion]: > """ > The version of TLS that has been negotiated on this connection. > """ > > > Cipher Suites > ~~~~~~~~~~~~~ > > Todo Cipher suites are going to be a mess! Too bad OpenSSL and GnuTLS do not use IANA names. :( It should be good enough to focus on a subset and use the wire protocol values as internal identifiers. https://raw.githubusercontent.com/tiran/tlsdb/master/tlsdb.json > > Protocol Negotiation > ~~~~~~~~~~~~~~~~~~~~ > > Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 > handshake. While NPN and ALPN are, at their fundamental level, built on top of > bytestrings, string-based APIs are frequently problematic as they allow for > errors in typing that can be hard to detect. > > For this reason, this module would define a type that protocol negotiation > implementations can pass and be passed. This type would wrap a bytestring, but > allow for aliases for well-known protocols. This allows us to avoid the > problems inherent in typos for well-known protocols, while allowing the full > extensibility of the protocol negotiation layer if needed. > > :: > > NextProtocol = namedtuple('NextProtocol', ['name']) > > H2 = NextProtocol(b'h2') > H2C = NextProtocol(b'h2c') > HTTP1 = NextProtocol(b'http/1.1') > WEBRTC = NextProtocol(b'webrtc') > C_WEBRTC = NextProtocol(b'c-webrtc') > FTP = NextProtocol(b'ftp') > STUN = NextProtocol(b'stun.nat-discovery') > TURN = NextProtocol(b'stun.turn') > > TLS Versions > ~~~~~~~~~~~~ > > It is often useful to be able to restrict the versions of TLS you're willing to > support. There are many security advantages in refusing to use old versions of > TLS, and some misbehaving servers will mishandle TLS clients advertising > support for newer versions. > > The following enumerated type can be used to gate TLS versions. Forward-looking > applications should almost never set a maximum TLS version unless they > absolutely must, as a TLS backend that is newer than the Python that uses it > may support TLS versions that are not in this enumerated type. > > Additionally, this enumerated type defines two additional flags that can always > be used to request either the lowest or highest TLS version supported by an > implementation. > > :: > > class TLSVersion(Enum): > MINIMUM_SUPPORTED > SSLv2 > SSLv3 > TLSv1 > TLSv1_1 > TLSv1_2 > TLSv1_3 > MAXIMUM_SUPPORTED > > > Errors > ~~~~~~ > > This module would define three base classes for use with error handling. Unlike > the other classes defined here, these classes are not *abstract*, as they have > no behaviour. They exist simply to signal certain common behaviours. Backends > should subclass these exceptions in their own packages, but needn't define any > behaviour for them. These exceptions should *never* be thrown directly, they > should always be subclassed. > > The definitions of the errors are below:: > > class TLSError(Exception): > """ > The base exception for all TLS related errors from any backend. > Catching this error should be sufficient to catch *all* TLS errors, > regardless of what backend is used. > """ > > class WantWriteError(TLSError): > """ > A special signaling exception used only when non-blocking or > buffer-only I/O is used. This error signals that the requested > operation cannot complete until more data is written to the network, > or until the output buffer is drained. > """ > > class WantReadError(TLSError): > """ > A special signaling exception used only when non-blocking or > buffer-only I/O is used. This error signals that the requested > operation cannot complete until more data is read from the network, or > until more data is available in the input buffer. > """ > > Changes to the Standard Library > =============================== > > The portions of the standard library that interact with TLS should be revised > to use these ABCs. This will allow them to function with other TLS backends. > This includes the following modules: > > - asyncio > - ftplib > - http.client > - imaplib > - nntplib > - poplib > - smtplib > > > Future > ====== > > Major future TLS features may require revisions of these ABCs. These revisions > should be made cautiously: many backends may not be able to move forward > swiftly, and will be invalidated by changes in these ABCs. This is acceptable, > but wherever possible features that are specific to individual implementations > should not be added to the ABCs. The ABCs should restrict themselves to > high-level descriptions of IETF-specified features. > > > ToDo > ==== > > * Consider adding a new parameter (``valid_subjects``?) to ``wrap_socket`` and > ``wrap_buffers`` that specifies in a *typed* manner what kind of entries in > the SAN field are acceptable. This would break the union between SNI and > cert validation, which may be a good thing (you can't SNI an IP address, but > you can validate a cert with one if you want). > * It's annoying that there's no type signature for fileobj. Do I really have to > define one as part of this PEP? Otherwise, how do I define the types of the > arguments to ``wrap_buffers``? > * Do we need ways to control hostname validation? > * Do we need to support getpeercert? Should we always return DER instead of the > weird semi-structured thing? > * How do we load certs from locations on disk? What about HSMs? > * How do we signal to load certs from the OS? What happens if an implementation > doesn't let you *not* load those certs? > > > References > ========== > > .. _ssl module: https://docs.python.org/3/library/ssl.html > .. _OpenSSL Library: https://www.openssl.org/ > .. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ > .. _certifi: https://pypi.org/project/certifi/ > .. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext > .. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket > .. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject > .. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError > > > Copyright > ========= > > This document has been placed in the public domain. > > > > _______________________________________________ > Security-SIG mailing list > Security-SIG at python.org > https://mail.python.org/mailman/listinfo/security-sig > From christian at cheimes.de Wed Jan 11 14:49:13 2017 From: christian at cheimes.de (Christian Heimes) Date: Wed, 11 Jan 2017 20:49:13 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> Message-ID: On 2017-01-11 20:01, Cory Benfield wrote: > Socket > ~~~~~~ > > The socket-wrapper ABC will be defined by the ``TLSWrappedSocket`` ABC, which > has the following definition:: > > class TLSWrappedSocket(metaclass=ABCMeta): > # The various socket methods all must be implemented. Their definitions > # have been elided from this class defintion in the PEP because they > # aren't instructive. I sent my first mail too early and forgot three things. Python's ssl module has additional weird modes. It's possible to create an unconnected SSLSocket and later connect it. Pseudo code: s1 = socket.socket() s2 = context.wrap_socket(s1) s2.connect((host, port)) AFAIK PyOpenSSL doesn't support this mode. How do we deal with unconnected sockets, UDP/DTLS and other transports? Are sockets limited to AF_INET / AF_INET6 and SOCK_STREAM? In the not-so-distant future SRV-ID validation will become relevant. In order to support dNSName, IPAddress, and SRV-ID validation, the TLS socket needs the hostname (if available), IP address, port and service type (e.g. http, ldap, xmpp-server, ...). For hostname validation we should also define how we are going to deal with encodings. Is the hostname always a IDN U-label, A-label or can it be both? Christian From cory at lukasa.co.uk Wed Jan 11 15:16:12 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 11 Jan 2017 20:16:12 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> Message-ID: <4948745C-B32C-4990-8FDD-922A7479EA59@lukasa.co.uk> > On 11 Jan 2017, at 19:36, Christian Heimes wrote: > > Since this function is only required for TLS servers and TLS client cert > authentication, I'd rather mark this function provisional or not define > it in the first version. Is that the best option? Will there be any TLS implementation that cannot meet a stripped down version of this function?s requirements? If we can achieve a minimum viable approach whereby 80% of the use-case (unencrypted/encrypted certs on disk) can be met, I?d like to be able to meet it. >> @abstractmethod >> def set_ciphers(self, ciphers: List[Ciphers]) -> None: >> """ >> Set the available ciphers for TLS connections created with this >> context. ``ciphers`` should be a list of ciphers from the >> ``Cipher`` registry. If none of the ``ciphers`` provided to this >> object are supported or available, a ``TLSError`` will be raised. >> """ > > Implementors should initial context with sensible default settings, > preferable system-wide settings. For example Fedora is currently > implementing https://fedoraproject.org/wiki/Changes/CryptoPolicy for > OpenSSL, NSS and GnuTLS. Is this an additional requirement to create_default_context, which should presumably be able using this API to do sensible things? >> @abstractmethod >> def wrap_buffers(self, incoming: Any, outgoing: Any, >> server_side=False: bool, >> server_hostname=None: Optional[str]) -> TLSWrappedBuffer: >> """ >> Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to >> create an in-memory stream for TLS. The SSL routines will read data >> from ``incoming`` and decrypt it, and write encrypted data to >> ``outgoing``. >> >> The ``server_side`` and ``server_hostname`` parameters have the >> same meaning as in ``wrap_socket``. >> """ > > > How about not defining this methods at all? IMO it makes no sense to > support client and server connections from the same context. This is > also the gist of a PEP I'm currently working on. When you say ?not defining this at all?, do you mean not using ?server_side?? Because we definitely need the some wrap methods. > Cipher suites are going to be a mess! Too bad OpenSSL and GnuTLS do not > use IANA names. :( It should be good enough to focus on a subset and use > the wire protocol values as internal identifiers. > > https://raw.githubusercontent.com/tiran/tlsdb/master/tlsdb.json Yeah, I was thinking an enum with values, which give us the advantage of a strongly-typed API as well as giving sensible internal identifiers. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Wed Jan 11 15:20:08 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 11 Jan 2017 20:20:08 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> Message-ID: > On 11 Jan 2017, at 19:49, Christian Heimes wrote: > > AFAIK PyOpenSSL doesn't support this mode. How do we deal with > unconnected sockets, UDP/DTLS and other transports? Are sockets limited > to AF_INET / AF_INET6 and SOCK_STREAM? To begin with, I think we need to restrict ourselves to SOCK_STREAM. There?s no need to get specific about address family I don?t think. As for unconnected sockets, I think I?d like to delay that concern unless someone wants to propose the API. > In the not-so-distant future SRV-ID validation will become relevant. In > order to support dNSName, IPAddress, and SRV-ID validation, the TLS > socket needs the hostname (if available), IP address, port and service > type (e.g. http, ldap, xmpp-server, ?). The advantage of this API is that it would be extensible. We can extend it as needed over time, we don?t need to shove everything in at once. So I?m inclined to want to defer this until we see what the implementations actually do. > For hostname validation we should also define how we are going to deal > with encodings. Is the hostname always a IDN U-label, A-label or can it > be both? Pass. I think you have a better idea of what is required here than I do: do you have a suggestion? From christian at cheimes.de Wed Jan 11 15:51:15 2017 From: christian at cheimes.de (Christian Heimes) Date: Wed, 11 Jan 2017 21:51:15 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <4948745C-B32C-4990-8FDD-922A7479EA59@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <4948745C-B32C-4990-8FDD-922A7479EA59@lukasa.co.uk> Message-ID: <183e5329-132c-a651-5227-2582ee9bd418@cheimes.de> On 2017-01-11 21:16, Cory Benfield wrote: > >> On 11 Jan 2017, at 19:36, Christian Heimes > > wrote: >> >> Since this function is only required for TLS servers and TLS client cert >> authentication, I'd rather mark this function provisional or not define >> it in the first version. > > Is that the best option? Will there be any TLS implementation that > cannot meet a stripped down version of this function?s requirements? If > we can achieve a minimum viable approach whereby 80% of the use-case > (unencrypted/encrypted certs on disk) can be met, I?d like to be able to > meet it. Of course we can solve any problem by adding just another layer of abstraction! :) In this case it seems preferable and matches my idea for 3.7 improvements. How about you change register_certificates() to accept a list of certificate instances (sorted from EE, intermediate 1 (issuer of EE), intermediate 2 (issuer of 1), intermediate 3 ...) and a key object. The exact nature of a certificate class is implementation specific. The approach has also the benefit that file to list of certificates is handled by the certificate class: class Certificate: @abstractclassmethod def from_buffer(cls, filename: (str, bytes)) -> List(Certificate): pass @abstractclassmethod def from_file(cls, filename: path) -> List(Certificate): pass >> Implementors should initial context with sensible default settings, >> preferable system-wide settings. For example Fedora is currently >> implementing https://fedoraproject.org/wiki/Changes/CryptoPolicy for >> OpenSSL, NSS and GnuTLS. > > Is this an additional requirement to create_default_context, which > should presumably be able using this API to do sensible things? I would rather make the default contexts secure by default without the need of a secure factory function. set_ciphers() should not be needed at all to get good settings. Only poor slobs should use set_ciphers() to enable bad ciphers for broken legacy systems. > When you say ?not defining this at all?, do you mean not using > ?server_side?? Because we definitely need the some wrap methods. Sorry, I was not precise here. I meant that both wrap methods should be limited to ClientContext and ServerContext classes. There is no good use case to have a context that supports both client and server connections. In Python code: del _BaseContext.wrap_socket del _BaseContext.wrap_buffers :) Christian From christian at cheimes.de Wed Jan 11 16:23:51 2017 From: christian at cheimes.de (Christian Heimes) Date: Wed, 11 Jan 2017 22:23:51 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> Message-ID: <85957ce9-fbd4-5551-858d-4ec09746ac4c@cheimes.de> On 2017-01-11 21:20, Cory Benfield wrote: > >> On 11 Jan 2017, at 19:49, Christian Heimes wrote: >> >> AFAIK PyOpenSSL doesn't support this mode. How do we deal with >> unconnected sockets, UDP/DTLS and other transports? Are sockets limited >> to AF_INET / AF_INET6 and SOCK_STREAM? > > To begin with, I think we need to restrict ourselves to SOCK_STREAM. There?s no need to get specific about address family I don?t think. As for unconnected sockets, I think I?d like to delay that concern unless someone wants to propose the API. * +1 for SOCK_STREAM * Although it's obvious to us, we should to state that IPAddress SAN validation is only defined for AF_INET and AF_INET6. * Do we need to define blocking / non blocking state of the socket? * What about STARTTLS support, do we need to define how to deal with data that has been transmitted before? Later: * unconnected socket and tlssocket.connect(). * Python's ssl module has unwrap (STOPTLS) to shutdown a TLS socket and return to non-encrypted communcation. > >> In the not-so-distant future SRV-ID validation will become relevant. In >> order to support dNSName, IPAddress, and SRV-ID validation, the TLS >> socket needs the hostname (if available), IP address, port and service >> type (e.g. http, ldap, xmpp-server, ?). > > The advantage of this API is that it would be extensible. We can extend it as needed over time, we don?t need to shove everything in at once. So I?m inclined to want to defer this until we see what the implementations actually do. Maybe we can get away without any extension to the API. SRV-ID uses underline to signal a service identifier, e.g. _http.www.example.org. In the future an application could pass down a service identifier as server_hostname argument and let the TLS library deal with it like this: if server_hostname.startswith('_'): srvid = server_hostname service, hostname = server_hostname.split('.', 1) service = service[1:] else: service = srvid = None hostname = server_hostname >> For hostname validation we should also define how we are going to deal >> with encodings. Is the hostname always a IDN U-label, A-label or can it >> be both? > > Pass. I think you have a better idea of what is required here than I do: do you have a suggestion? OpenSSL's hostname verification code accept IDN A-labels. SubjecAltNames are encoded as IA5String containing IDN A-labels, too. For most flexibility the wrap functions should accept server_hostnames as both U-label and A-label. If you add an encode_hostname() method to the context, than users can easily override the method to switch between IDNA 2003, 2008 or perform custom checks to prevent homoglyphic confusion attacks. class _BaseContext: def encode_hostname(self, hostname: str) -> str: """Convert IDN U-label to A-label """ @abstractmethod def wrap_socket(self, socket: socket.socket, auto_handshake=True: bool, server_hostname=None: Optional[str]): if server_hostname is not None: server_hostname = self.context.encode_hostname( server_hostname) Christian From wes.turner at gmail.com Thu Jan 12 00:36:21 2017 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 11 Jan 2017 23:36:21 -0600 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> Message-ID: On Wednesday, January 11, 2017, Christian Heimes wrote: On 2017-01-11 20:01, Cory Benfield wrote: > > > The ``Context`` abstract base class defines an object that allows > configuration > > of TLS. It can be thought of as a factory for ``TLSWrappedSocket`` and > > ``TLSWrappedBuffer`` objects. > > > > The ``Context`` abstract base class has the following class definition:: > > > > TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] > > ServerNameCallback = Callable[[TLSBufferObject, Optional[str], > Context], Any] > > > > class _BaseContext(metaclass=ABCMeta): > > > > @property > > @abstractmethod > > def validate_certificates(self) -> bool: > > """ > > Whether to validate the TLS certificates. This switch > operates at a > > very broad scope: either validation is enabled, in which > case all > > forms of validation are performed including hostname > validation if > > possible, or validation is disabled, in which case no > validation is > > performed. > > > > Not all backends support having their certificate validation > > disabled. If a backend does not support having their > certificate > > validation disabled, attempting to set this property to > ``False`` > > will throw a ``TLSError``. > > """ > > > > @validate_certificates.setter > > @abstractmethod > > def validate_certificates(self, val: bool) -> None: > > pass > > > For 3.7 I'm planning to replace ssl.match_hostname() with OpenSSL > 1.0.2's API. For now the one flag is enough. Later we can discuss > settings for wildcard, IP address and CN matching. > > > > > @abstractmethod > > def register_certificates(self, > > certificates: str, > > key=None: Optional[str], > > password=None: Optional[Callable[[], > Union[AnyStr, bytearray]]]) -> None: > > """ > > Loads a certificate, a number of intermediate certificates, > and the > > corresponding private key. These certificates will be > offered to > > the remote peer during the handshake if required. > > > > The ``certificates`` argument must be a bytestring > containing the > > PEM-encoded certificates. The first PEM-encoded certificate > must be > > the leaf certificate. All subsequence certificates will be > offered > > as intermediate additional certificates. > > > > The ``key`` argument, if present, must contain the > PEM-encoded > > private key associated with the leaf certificate. If not > present, > > the private key will be extracted from ``certificates``. > > > > The ``password`` argument may be a function to call to get > the > > password for decrypting the private key. It will only be > called if > > the private key is encrypted and a password is necessary. It > will > > be called with no arguments, and it should return a string, > bytes, > > or bytearray. If the return value is a string it will be > encoded as > > UTF-8 before using it to decrypt the key. Alternatively a > string, > > bytes, or bytearray value may be supplied directly as the > password > > argument. It will be ignored if the private key is not > encrypted > > and no password is needed. > > """ > > I don't think this function works for all libraries and use cases. For > some implementations the order of certificates is very important. For > NSS and PKCS#11 we rather need to specify the slot or nick name of the > cert. For 3.7 I also like to introduce X509 objects and EVP_Key wrapper, > so this function would need to consume a stack of certificates. > > Since this function is only required for TLS servers and TLS client cert > authentication, I'd rather mark this function provisional or not define > it in the first version. > > > This may be a bit of a different use case (and possibly worth having in the first version of a new tls module): "Hitless TLS Certificate Rotation in Go" https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/ - Can/could this be done with only set_sni_callback ? - VerifyPeerCertificate https://github.com/golang/go/issues/16363 - https://github.com/golang/go/blob/release-branch.go1.8/ src/crypto/tls/common.go#L406 > > > @abstractmethod > > def set_ciphers(self, ciphers: List[Ciphers]) -> None: > > """ > > Set the available ciphers for TLS connections created with > this > > context. ``ciphers`` should be a list of ciphers from the > > ``Cipher`` registry. If none of the ``ciphers`` provided to > this > > object are supported or available, a ``TLSError`` will be > raised. > > """ > > Implementors should initial context with sensible default settings, > preferable system-wide settings. For example Fedora is currently > implementing https://fedoraproject.org/wiki/Changes/CryptoPolicy for > OpenSSL, NSS and GnuTLS. > > > > > > > @abstractmethod > > def set_inner_protocols(self, protocols: List[NextProtocol]) -> > None: > > """ > > Specify which protocols the socket should advertise as > supported > > during the TLS handshake. This may be advertised using > either or > > both of ALPN or NPN. > > > > ``protocols`` should be a list of acceptable protocols in > the form > > of ``NextProtocol`` objects, such as ``[H2, HTTP1]``, > ordered by > > preference. The selection of the protocol will happen during > the > > handshake, and will use whatever protocol negotiation > mechanisms > > are available and supported by both peers. > > > > If the TLS implementation doesn't support protocol > negotiation, > > this method will raise ``NotImplementedError``. > > """ > > > > @abstractmethod > > def set_sni_callback(self, callback: > Optional[ServerNameCallback]) -> None: > > """ > > Register a callback function that will be called after the > TLS > > Client Hello handshake message has been received by the TLS > server > > when the TLS client specifies a server name indication. > > > > Only one callback can be set per ``Context``. If > ``callback`` is > > ``None`` then the callback is disabled. Calling this > function a > > subsequent time will disable the previously registered > callback. > > > > The ``callback`` function will be called with three > arguments: the > > first will be the ``TLSBufferObject`` for the connection; the > > second will be a string that represents the server name that > the > > client is intending to communicate (or ``None`` if the TLS > Client > > Hello does not contain a server name); and the third > argument will > > be the original ``Context``. The server name argument will > be the > > IDNA *decoded* server name. > > > > The ``callback`` must return ``None`` to allow negotiation to > > continue. Other return values signal errors. Attempting to > control > > what error is signaled by the underlying TLS implementation > is not > > specified in this API, but is up to the concrete > implementation to > > handle. > > """ > > > > @abstractmethod > > def set_version_range(self, lower_bound=None: > Optional[TLSVersion], > > upper_bound=None: Optional[TLSVersion]) -> > None: > > """ > > Set the minumum and maximum versions of TLS that should be > allowed > > on TLS connections made by this context. > > > > If present, ``lower_bound`` will set the lowest acceptable > TLS > > version. If present, ``upper_bound`` will set the highest > > acceptable TLS version. If either argument is ``None``, this > will > > leave that bound unchanged. > > """ > So, with ``` class TLSVersion(Enum): MINIMUM_SUPPORTED SSLv2 SSLv3 TLSv1 TLSv1_1 TLSv1_2 TLSv1_3 MAXIMUM_SUPPORTED ``` What are the (signed?) integer values? -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Jan 12 03:26:21 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 12 Jan 2017 08:26:21 +0000 Subject: [Security-sig] new TLS module In-Reply-To: <58768920.6040701@stoneleaf.us> References: <58768920.6040701@stoneleaf.us> Message-ID: > On 11 Jan 2017, at 19:36, Ethan Furman wrote: > > With TLSv1.2 becoming mandatory in the near future and the various issues with the ssl module there is more interest in a tls module for Python. > > What should such a module include? What should it not include? Any reason to not also have a backport of this module? Ethan, You may find the discussion regarding a draft PEP I?ve been working on worth a read before you charge too far down this road. Cory From cory at lukasa.co.uk Thu Jan 12 03:45:20 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 12 Jan 2017 08:45:20 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <85957ce9-fbd4-5551-858d-4ec09746ac4c@cheimes.de> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <85957ce9-fbd4-5551-858d-4ec09746ac4c@cheimes.de> Message-ID: <8E82A7D9-EF98-4508-BAB7-D6CB94D417EB@lukasa.co.uk> > On 11 Jan 2017, at 21:23, Christian Heimes wrote: > > * Do we need to define blocking / non blocking state of the socket? I think we want to support both. I?m not sure we need to expose it in the API: the implementation can check by asking the socket directly if it cares. > * What about STARTTLS support, do we need to define how to deal with > data that has been transmitted before? Hrm. I?m inclined to want to hold off on that to begin with, though you?re welcome to propose an API extension if you have a concrete idea of how to do it. > Maybe we can get away without any extension to the API. SRV-ID uses > underline to signal a service identifier, e.g. _http.www.example.org. In > the future an application could pass down a service identifier as > server_hostname argument and let the TLS library deal with it like this: > > if server_hostname.startswith('_'): > srvid = server_hostname > service, hostname = server_hostname.split('.', 1) > service = service[1:] > else: > service = srvid = None > hostname = server_hostname Yup. That could well work. Another good argument for ?wait and see? ;). > OpenSSL's hostname verification code accept IDN A-labels. SubjecAltNames > are encoded as IA5String containing IDN A-labels, too. For most > flexibility the wrap functions should accept server_hostnames as both > U-label and A-label. If you add an encode_hostname() method to the > context, than users can easily override the method to switch between > IDNA 2003, 2008 or perform custom checks to prevent homoglyphic > confusion attacks. My question here is: do we need to pursue this in the abstract API? Or can it be left up to concrete implementations to worry about? Cory From cory at lukasa.co.uk Thu Jan 12 03:47:39 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 12 Jan 2017 08:47:39 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> Message-ID: > On 11 Jan 2017, at 21:44, Wes Turner wrote: > > This may be a bit of a different use case (and possibly worth having in the first version of a new tls module): > > "Hitless TLS Certificate Rotation in Go" > https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/ > > - Can/could this be done with only set_sni_callback ? Yes, it can be. Twisted has an extension module, txsni, that uses the SNI callback to choose which certificate to provide. This is basically identical to the Go GetCertificate callback function. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Jan 12 03:49:09 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 12 Jan 2017 08:49:09 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> Message-ID: <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> > On 12 Jan 2017, at 05:36, Wes Turner wrote: > > So, with > > ``` > class TLSVersion(Enum): > MINIMUM_SUPPORTED > SSLv2 > SSLv3 > TLSv1 > TLSv1_1 > TLSv1_2 > TLSv1_3 > MAXIMUM_SUPPORTED > ``` > > What are the (signed?) integer values? Sorry Wes, I missed this in my first reply. Does it matter? The concrete implementations should be looking for the enum entries directly. The actual values shouldn?t matter, especially as different TLS APIs have very different methods of actually configuring this. Cory From christian at cheimes.de Thu Jan 12 05:14:41 2017 From: christian at cheimes.de (Christian Heimes) Date: Thu, 12 Jan 2017 11:14:41 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <8E82A7D9-EF98-4508-BAB7-D6CB94D417EB@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <85957ce9-fbd4-5551-858d-4ec09746ac4c@cheimes.de> <8E82A7D9-EF98-4508-BAB7-D6CB94D417EB@lukasa.co.uk> Message-ID: On 2017-01-12 09:45, Cory Benfield wrote: > >> On 11 Jan 2017, at 21:23, Christian Heimes wrote: >> >> * Do we need to define blocking / non blocking state of the socket? > > I think we want to support both. I?m not sure we need to expose it in the API: the implementation can check by asking the socket directly if it cares. Do we need to support both? I thought that wrap_buffers() is the preferred way for non-blocking TLS with an event loop. You are the expert, do SChannel and SecureTransport work with non-blocking sockets and WantRead/WantWrite? > >> * What about STARTTLS support, do we need to define how to deal with >> data that has been transmitted before? > > Hrm. I?m inclined to want to hold off on that to begin with, though you?re welcome to propose an API extension if you have a concrete idea of how to do it. Not really, just add it to the ToDo pile :) >> OpenSSL's hostname verification code accept IDN A-labels. SubjecAltNames >> are encoded as IA5String containing IDN A-labels, too. For most >> flexibility the wrap functions should accept server_hostnames as both >> U-label and A-label. If you add an encode_hostname() method to the >> context, than users can easily override the method to switch between >> IDNA 2003, 2008 or perform custom checks to prevent homoglyphic >> confusion attacks. > > My question here is: do we need to pursue this in the abstract API? Or can it be left up to concrete implementations to worry about? I see some merit to either have a formal encoding hook on the context or to limit server_hostname to punycode. But before we decide, we need to study TLS implementations. OpenSSL requires IDN A-labels. NSS' CERT_VerifyCertName() consumes IDNA, too. InitializeSecurityContext() pszTargetName has no encoding specification. I'm not familiar with SecureTransport. Christian From cory at lukasa.co.uk Thu Jan 12 05:26:40 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 12 Jan 2017 10:26:40 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <85957ce9-fbd4-5551-858d-4ec09746ac4c@cheimes.de> <8E82A7D9-EF98-4508-BAB7-D6CB94D417EB@lukasa.co.uk> Message-ID: <11839D03-192E-46A5-A86D-A00FDD815957@lukasa.co.uk> > On 12 Jan 2017, at 10:14, Christian Heimes wrote: > > On 2017-01-12 09:45, Cory Benfield wrote: >> >>> On 11 Jan 2017, at 21:23, Christian Heimes wrote: >>> >>> * Do we need to define blocking / non blocking state of the socket? >> >> I think we want to support both. I?m not sure we need to expose it in the API: the implementation can check by asking the socket directly if it cares. > > Do we need to support both? I thought that wrap_buffers() is the > preferred way for non-blocking TLS with an event loop. You are the > expert, do SChannel and SecureTransport work with non-blocking sockets > and WantRead/WantWrite? Preferred is a general truth, but not always specifically true. In particular, there is value in non-blocking sockets outwith an event loop. Calling me ?the expert? on this topic is a little strong, but I know that SecureTransport does. Cory From ncoghlan at gmail.com Thu Jan 12 07:37:14 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 12 Jan 2017 22:37:14 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> Message-ID: On 12 January 2017 at 18:49, Cory Benfield wrote: > >> On 12 Jan 2017, at 05:36, Wes Turner wrote: >> >> So, with >> >> ``` >> class TLSVersion(Enum): >> MINIMUM_SUPPORTED >> SSLv2 >> SSLv3 >> TLSv1 >> TLSv1_1 >> TLSv1_2 >> TLSv1_3 >> MAXIMUM_SUPPORTED >> ``` >> >> What are the (signed?) integer values? > > Sorry Wes, I missed this in my first reply. > > Does it matter? The concrete implementations should be looking for the enum entries directly. The actual values shouldn?t matter, especially as different TLS APIs have very different methods of actually configuring this. For "Enums" where I genuinely don't care about the values, I'll typically set them to the string that matches the attribute name: ``` class TLSVersion(Enum): SSLv2 = "SSLv2" MINIMUM_SUPPORTED = SSLv2 SSLv3 = "SSLv3" TLSv1= "TLSv1" TLSv1_1 = "TLSv1_1" TLSv1_2 = "TLSv1_2" TLSv1_3 = "TLSv1_3" MAXIMUM_SUPPORTED = TLSv1_3 ``` That way folks get sensible answers regardless of whether they reference the enum entry name or its value, or render it directly with repr() or str(): >>> TLSVersion.MINIMUM_SUPPORTED >>> TLSVersion.MAXIMUM_SUPPORTED >>> str(TLSVersion.MAXIMUM_SUPPORTED) 'TLSVersion.TLSv1_3' >>> TLSVersion.MAXIMUM_SUPPORTED.name 'TLSv1_3' >>> TLSVersion.MAXIMUM_SUPPORTED.value 'TLSv1_3' Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Thu Jan 12 07:44:34 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 12 Jan 2017 12:44:34 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> Message-ID: > On 12 Jan 2017, at 12:37, Nick Coghlan wrote: > > For "Enums" where I genuinely don't care about the values, I'll > typically set them to the string that matches the attribute name: > > ``` > class TLSVersion(Enum): > SSLv2 = "SSLv2" > MINIMUM_SUPPORTED = SSLv2 > SSLv3 = "SSLv3" > TLSv1= "TLSv1" > TLSv1_1 = "TLSv1_1" > TLSv1_2 = "TLSv1_2" > TLSv1_3 = "TLSv1_3" > MAXIMUM_SUPPORTED = TLSv1_3 > ``` > > That way folks get sensible answers regardless of whether they > reference the enum entry name or its value, or render it directly with > repr() or str(): > >>>> TLSVersion.MINIMUM_SUPPORTED > >>>> TLSVersion.MAXIMUM_SUPPORTED > >>>> str(TLSVersion.MAXIMUM_SUPPORTED) > 'TLSVersion.TLSv1_3' >>>> TLSVersion.MAXIMUM_SUPPORTED.name > 'TLSv1_3' >>>> TLSVersion.MAXIMUM_SUPPORTED.value > 'TLSv1_3' We can do that. I should note that MINIMUM_SUPPORTED and MAXIMUM_SUPPORTED are not intended to be equal to SSLv2 and TLSv1_3, or indeed to any other value in this enum. They are future-proofing tools that allow users to say ?I want TLSv1 *or higher*? without setting an upper bound on what ?higher? means. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jan 12 09:09:51 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Jan 2017 00:09:51 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> Message-ID: On 12 January 2017 at 22:44, Cory Benfield wrote: > We can do that. > > I should note that MINIMUM_SUPPORTED and MAXIMUM_SUPPORTED are not intended > to be equal to SSLv2 and TLSv1_3, or indeed to any other value in this enum. > They are future-proofing tools that allow users to say ?I want TLSv1 *or > higher*? without setting an upper bound on what ?higher? means. Cool, I wasn't sure how you intended to handle that, and supplying the values will make it explicit that those are really only useful in "version_range" and not anywhere else. Although at that point the question becomes whether or not they're offering any benefit beyond just using "None" in the appropriate location. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at cheimes.de Thu Jan 12 09:24:30 2017 From: christian at cheimes.de (Christian Heimes) Date: Thu, 12 Jan 2017 15:24:30 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> Message-ID: On 2017-01-12 15:09, Nick Coghlan wrote: > On 12 January 2017 at 22:44, Cory Benfield wrote: >> We can do that. >> >> I should note that MINIMUM_SUPPORTED and MAXIMUM_SUPPORTED are not intended >> to be equal to SSLv2 and TLSv1_3, or indeed to any other value in this enum. >> They are future-proofing tools that allow users to say ?I want TLSv1 *or >> higher*? without setting an upper bound on what ?higher? means. > > Cool, I wasn't sure how you intended to handle that, and supplying the > values will make it explicit that those are really only useful in > "version_range" and not anywhere else. Although at that point the > question becomes whether or not they're offering any benefit beyond > just using "None" in the appropriate location. I have a working PoC patch for a TLS version enum and set version range method on https://bugs.python.org/issue27876. We have to consider different kinds of min and max version: 1) min and max offical SSL/TLS standards 2) min and max supported standards for a TLS implementation 3) min and max version we consider as secure enough As of now (1) is SSLv2 / TLSv1.2 because TLSv1.3 is still a draft. (2) depends on the library, it is SSLv3 / TLSv1.2 for OpenSSL and SSLv3 / TLSv1.3 for NSS on the client side and SSLv3 / TLSv1.2 for NSS on the server side for default builds. (3) is TLSv1.0 and max of (2). Contrary to my PoC we should also differentiate between MAXIMUM_SUPPORTED and whatever the maximum supported TLS version for a TLS implementation is. For example set_version_range(max=MAXIMUM_SUPPORTED) should never fail but set_version_range(max=TLS_1_3) should fail for OpenSSL. Christian From wes.turner at gmail.com Thu Jan 12 12:07:22 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 12 Jan 2017 11:07:22 -0600 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> Message-ID: On Thursday, January 12, 2017, Cory Benfield wrote: > > On 11 Jan 2017, at 21:44, Wes Turner > wrote: > > This may be a bit of a different use case (and possibly worth having in > the first version of a new tls module): > > "Hitless TLS Certificate Rotation in Go" > https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/ > > - Can/could this be done with only set_sni_callback ? > > > Yes, it can be. Twisted has an extension module, txsni, that uses the SNI > callback to choose which certificate to provide. > https://github.com/glyph/txsni > This is basically identical to the Go GetCertificate callback function. > There's more config than just the cert, though. There are really two reqs mentioned: SNI (Server Name Indication), and "hot" TLS config detection/selection: """ The idea is to allow an administrator to force the whole cluster to migrate away from an old root CA transparently, removing its existence from the trust stores of all the nodes participating in the Swarm. This means that we need control over the whole TLS config, instead of controlling only which certificate is currently being served. """ '"" We chose to create a MutableTLSCreds struct, which implements this TransportCredentials interface and allows the caller to simply change the TLS Config by calling LoadNewTLSConfig. """ IIUC, we'd currently have to create a new context to change any config other than the cert? > > Cory > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Jan 12 12:16:34 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 12 Jan 2017 11:16:34 -0600 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> Message-ID: On Thursday, January 12, 2017, Christian Heimes wrote: > On 2017-01-12 15:09, Nick Coghlan wrote: > > On 12 January 2017 at 22:44, Cory Benfield > wrote: > >> We can do that. > >> > >> I should note that MINIMUM_SUPPORTED and MAXIMUM_SUPPORTED are not > intended > >> to be equal to SSLv2 and TLSv1_3, or indeed to any other value in this > enum. > >> They are future-proofing tools that allow users to say ?I want TLSv1 *or > >> higher*? without setting an upper bound on what ?higher? means. > > > > Cool, I wasn't sure how you intended to handle that, and supplying the > > values will make it explicit that those are really only useful in > > "version_range" and not anywhere else. Although at that point the > > question becomes whether or not they're offering any benefit beyond > > just using "None" in the appropriate location. > > > I have a working PoC patch for a TLS version enum and set version range > method on https://bugs.python.org/issue27876. ``` def __init__(self, prettyname, wireprotocol, offset): self.prettyname = prettyname self.wireprotocol = wireprotocol self.noflag = OP_NO_FLAGS[offset] self.minflag = sum(OP_NO_FLAGS[:offset]) self.maxflag = sum(OP_NO_FLAGS[offset+1:]) ``` - Do these need a __cmp__()? - Are there concrete-implementation-specific const constants for each library? > We have to consider different kinds of min and max version: > > 1) min and max offical SSL/TLS standards > 2) min and max supported standards for a TLS implementation > 3) min and max version we consider as secure enough The Apache HTTPD `SSLProtocol` and Nginx `ssl_protocols` options support different methods of whitelisting and blacklisting. https://mozilla.github.io/server-side-tls/ssl-config-generator/ modern (2017): - SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 - ssl_protocols TLSv1.2; > > As of now (1) is SSLv2 / TLSv1.2 because TLSv1.3 is still a draft. > (2) depends on the library, it is SSLv3 / TLSv1.2 for OpenSSL and SSLv3 > / TLSv1.3 for NSS on the client side and SSLv3 / TLSv1.2 for NSS on the > server side for default builds. > > (3) is TLSv1.0 and max of (2). > > Contrary to my PoC we should also differentiate between > MAXIMUM_SUPPORTED and whatever the maximum supported TLS version for a > TLS implementation is. For example > set_version_range(max=MAXIMUM_SUPPORTED) should never fail but > set_version_range(max=TLS_1_3) should fail for OpenSSL. > > Christian > _______________________________________________ > Security-SIG mailing list > Security-SIG at python.org > https://mail.python.org/mailman/listinfo/security-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at cheimes.de Thu Jan 12 14:13:53 2017 From: christian at cheimes.de (Christian Heimes) Date: Thu, 12 Jan 2017 20:13:53 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> Message-ID: <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> On 2017-01-12 18:07, Wes Turner wrote: > > > On Thursday, January 12, 2017, Cory Benfield > wrote: > > >> On 11 Jan 2017, at 21:44, Wes Turner > > wrote: >> >> This may be a bit of a different use case (and possibly worth >> having in the first version of a new tls module): >> >> "Hitless TLS Certificate Rotation in Go" >> https://diogomonica.com/2017/01/11/hitless-tls-certificate-rotation-in-go/ >> >> >> - Can/could this be done with only set_sni_callback ? > > Yes, it can be. Twisted has an extension module, txsni, that uses > the SNI callback to choose which certificate to provide. > > > https://github.com/glyph/txsni > > > This is basically identical to the Go GetCertificate callback function. > > > There's more config than just the cert, though. There are really two > reqs mentioned: SNI (Server Name Indication), and "hot" TLS config > detection/selection: > > """ > The idea is to allow an administrator to force the whole cluster to > migrate away from an old root CA transparently, removing its existence > from the trust stores of all the nodes participating in the Swarm. This > means that we need control over the whole TLS config, instead of > controlling only which certificate is currently being served. > """ > '"" > We chose to create a MutableTLSCreds > struct, > which implements this TransportCredentials > interface and > allows the caller to simply change the TLS Config by > calling |LoadNewTLSConfig|. > """ > > IIUC, we'd currently have to create a new context to change any config > other than the cert? Such advanced features are out-of-scope for the PEP. Please remember that any additional features makes it way more complicated for implementers. Personally I would rather remove half of the PEP than add new things. For pip, requests, and similar libraries only a subset of the features are really required: * ClientContext.wrap_socket() * a function like create_default_context() to create a client context that either loads a CA file or the system's trust store * TLSError * TLSWrappedSocket with socket.socket API Cory likes to define the API the additional features and server-side TLS. I consider it useful, partly advanced and nice-to-have, but not mandatory to solve the immediate issue we are facing now. [1] Let's keep it simple. We can always define an enhanced superset of the TLS ABC later. But we cannot remove features or change API in an incompatible way later. Christian https://mail.python.org/pipermail/distutils-sig/2017-January/029970.html From christian at cheimes.de Thu Jan 12 14:26:29 2017 From: christian at cheimes.de (Christian Heimes) Date: Thu, 12 Jan 2017 20:26:29 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <2D591DAC-C2E5-433B-90D9-18221A6E270D@lukasa.co.uk> Message-ID: <8b72e0de-3751-f855-1224-dd325ea0d9f3@cheimes.de> On 2017-01-12 18:16, Wes Turner wrote: > - Do these need a __cmp__()? > - Are there concrete-implementation-specific const constants for each > library? Why would you need to compare these? The TLS API will merely provide generic constants. Every implementation of the unified TLS API needs to map the constants somehow. I assume that TLSVersion enum is going to have a human readable name and a machine readable constants, preferable the wire protocol constant. https://bugs.python.org/issue27876 is just a PoC for OpenSSL and Python's ssl module. > The Apache HTTPD `SSLProtocol` and Nginx `ssl_protocols` options support > different methods of whitelisting and blacklisting. > > https://mozilla.github.io/server-side-tls/ssl-config-generator/ modern > (2017): > > - SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 > - ssl_protocols TLSv1.2; Nit picking mode: Apache HTTPD does not have a SSLProtocol option. mod_ssl implements the SSLProtocol option. Other TLS libraries for Apache (mod_nss) have different options, though. Some TLS libraries only support min and max TLS version, not version picking like OpenSSL. It doesn't make much sense to support TLS 1.0 and 1.2 but disallow TLS 1.1, too. A version range with secure default settings is both the simplest and most generic approach. Christian From donald at stufft.io Thu Jan 12 14:29:01 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 12 Jan 2017 14:29:01 -0500 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> Message-ID: <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> > On Jan 12, 2017, at 2:13 PM, Christian Heimes wrote: > > Let's keep it simple. We can always define an enhanced superset of the > TLS ABC later. But we cannot remove features or change API in an > incompatible way later. I think the server side stuff makes sense, it?ll be important for projects like Twisted and such and isn?t really *that* much more effort. Getting too lost in the weeds over advanced features like hot-config-reload I agree is a bad use of resources. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Thu Jan 12 14:39:59 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 12 Jan 2017 19:39:59 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> Message-ID: <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> I agree with Christian and Donald (unsurprisingly). The key thing to note is that we can extend this API as time goes on and we get a better understanding of what's happening. And any application that is doing hot TLS config changes is likely not going to be agnostic to the concrete TLS implementation it uses anyway, given that many implementations won't be sensibly able to do it. I'm not even sure about the specific API we're using for SNI: I might just want to restrict it to emitting new certificates. Cory > On 12 Jan 2017, at 19:29, Donald Stufft wrote: > > >> On Jan 12, 2017, at 2:13 PM, Christian Heimes wrote: >> >> Let's keep it simple. We can always define an enhanced superset of the >> TLS ABC later. But we cannot remove features or change API in an >> incompatible way later. > > > I think the server side stuff makes sense, it?ll be important for projects like Twisted and such and isn?t really *that* much more effort. Getting too lost in the weeds over advanced features like hot-config-reload I agree is a bad use of resources. > > ? > Donald Stufft > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jan 12 14:46:21 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 12 Jan 2017 14:46:21 -0500 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: > On Jan 12, 2017, at 2:39 PM, Cory Benfield wrote: > > I'm not even sure about the specific API we're using for SNI: I might just want to restrict it to emitting new certificates. I am pro restricting the API, can always relax restrictions later. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Jan 12 14:36:54 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 12 Jan 2017 13:36:54 -0600 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> Message-ID: On Thursday, January 12, 2017, Christian Heimes wrote: > On 2017-01-12 18:07, Wes Turner wrote: > > > > > > On Thursday, January 12, 2017, Cory Benfield > > >> wrote: > > > > > >> On 11 Jan 2017, at 21:44, Wes Turner > >> ');>> > wrote: > >> > >> This may be a bit of a different use case (and possibly worth > >> having in the first version of a new tls module): > >> > >> "Hitless TLS Certificate Rotation in Go" > >> https://diogomonica.com/2017/01/11/hitless-tls-certificate- > rotation-in-go/ > >> rotation-in-go/> > >> > >> - Can/could this be done with only set_sni_callback ? > > > > Yes, it can be. Twisted has an extension module, txsni, that uses > > the SNI callback to choose which certificate to provide. > > > > > > https://github.com/glyph/txsni > > > > > > This is basically identical to the Go GetCertificate callback > function. > > > > > > There's more config than just the cert, though. There are really two > > reqs mentioned: SNI (Server Name Indication), and "hot" TLS config > > detection/selection: > > > > """ > > The idea is to allow an administrator to force the whole cluster to > > migrate away from an old root CA transparently, removing its existence > > from the trust stores of all the nodes participating in the Swarm. This > > means that we need control over the whole TLS config, instead of > > controlling only which certificate is currently being served. > > """ > > '"" > > We chose to create a MutableTLSCreds > > struct, > > which implements this TransportCredentials > > interface and > > allows the caller to simply change the TLS Config by > > calling |LoadNewTLSConfig|. > > """ > > > > IIUC, we'd currently have to create a new context to change any config > > other than the cert? > > Such advanced features are out-of-scope for the PEP. Please remember > that any additional features makes it way more complicated for > implementers. Personally I would rather remove half of the PEP than add > new things. For pip, requests, and similar libraries only a subset of > the features are really required: > > * ClientContext.wrap_socket() > * a function like create_default_context() to create a client context > that either loads a CA file or the system's trust store > * TLSError > * TLSWrappedSocket with socket.socket API > > Cory likes to define the API the additional features and server-side > TLS. I consider it useful, partly advanced and nice-to-have, but not > mandatory to solve the immediate issue we are facing now. [1] > > Let's keep it simple. We can always define an enhanced superset of the > TLS ABC later. But we cannot remove features or change API in an > incompatible way later. I understand your scope concern. In order to accomplish the aforementioned "hot" TLS config selection, - Is there context-state to pass to TLSWrappedSocket.do_handshake(self, [context/configurator])? - Or, would it be possible to specify injection so as to make subclassing unnecessary? - Would it then be preferable to use a unified configuration object with AttributeErrors and ValueErrors for the context (and any potential connection-specific reconfigurations); instead of kwargs here and there? > > Christian > > https://mail.python.org/pipermail/distutils-sig/2017-January/029970.html > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sigmavirus24 at gmail.com Thu Jan 12 15:36:14 2017 From: sigmavirus24 at gmail.com (Ian Cordasco) Date: Thu, 12 Jan 2017 12:36:14 -0800 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: -----Original Message----- From:?Donald Stufft Reply:?Donald Stufft Date:?January 12, 2017 at 13:46:26 To:?Cory Benfield Cc:?security-sig at python.org , Christian Heimes Subject:? Re: [Security-sig] Unified TLS API for Python > > > On Jan 12, 2017, at 2:39 PM, Cory Benfield wrote: > > > > I'm not even sure about the specific API we're using for SNI: I might just want to restrict > it to emitting new certificates. > > > I am pro restricting the API, can always relax restrictions later. Expanding APIs is always leagues easier than contracting them. Starting off with the minimum necessary API makes perfect sense. As needs are found that it cannot meet, then expanding it slowly and methodically will be easy and painless. In other words, +1 on keeping it small to start and restricting the API. -- Ian Cordasco From wes.turner at gmail.com Thu Jan 12 16:05:10 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 12 Jan 2017 15:05:10 -0600 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: +1 for start simple and iterate; but expecting a config object is not easy to add later. class SSLConfig(dict): def validate(self): """check types and (implementation-specific) ranges""" class _BaseContext(metaclass=ABCMeta): # def validate_config(self, cfg: Union[Dict, SSLConfig]) -> Boolean: """check types and (implementation-specific) ranges""" if not hasattr(cfg, 'validate'): cfg = SSLConfig(cfg) cfg.validate() self.cfg = cfg || return cfg def set_config(self, cfg: Dict): self.register_certificates(cfg[*]) self.set_ciphers(cfg[*]) self.set_inner_protocls(cfg[*]) if cfg.get(*) is not None: self.set_sni_callback(cfg[*]) Why a configuration object makes sense here: - Centralized config validation - Implementations may have unique parameters - Convenience: just pass the config params from {JSON, TOML, YAML, configobj, PasteDeploy, pyramid, ...} with sane validation On Thu, Jan 12, 2017 at 2:36 PM, Ian Cordasco wrote: > -----Original Message----- > From: Donald Stufft > Reply: Donald Stufft > Date: January 12, 2017 at 13:46:26 > To: Cory Benfield > Cc: security-sig at python.org , Christian > Heimes > Subject: Re: [Security-sig] Unified TLS API for Python > > > > > > On Jan 12, 2017, at 2:39 PM, Cory Benfield wrote: > > > > > > I'm not even sure about the specific API we're using for SNI: I might > just want to restrict > > it to emitting new certificates. > > > > > > I am pro restricting the API, can always relax restrictions later. > > Expanding APIs is always leagues easier than contracting them. > Starting off with the minimum necessary API makes perfect sense. As > needs are found that it cannot meet, then expanding it slowly and > methodically will be easy and painless. > > In other words, +1 on keeping it small to start and restricting the API. > -- > Ian Cordasco > _______________________________________________ > Security-SIG mailing list > Security-SIG at python.org > https://mail.python.org/mailman/listinfo/security-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jan 13 01:09:45 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Jan 2017 16:09:45 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: On 13 January 2017 at 07:05, Wes Turner wrote: > +1 for start simple and iterate; > but expecting a config object is not easy to add later. Yes, it is - all that is necessary is to add a "make_ssl_context" helper function that translates from the declarative configuration format (however defined) to the programmatic API and returns a configured context of the requested type. The appropriate time to define that lowest-common-denominator configuration format is *after* there is a working programmatic API that covers at least the 3 major implementations of interest (OpenSSL, SecureTransport, SChannel), and hopefully a few other implementations as well (e.g. NSS, BoringSSL). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Fri Jan 13 01:32:26 2017 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 13 Jan 2017 00:32:26 -0600 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: On Fri, Jan 13, 2017 at 12:09 AM, Nick Coghlan wrote: > On 13 January 2017 at 07:05, Wes Turner wrote: > > +1 for start simple and iterate; > > but expecting a config object is not easy to add later. > > Yes, it is - all that is necessary is to add a "make_ssl_context" > helper function that translates from the declarative configuration > format (however defined) to the programmatic API and returns a > configured context of the requested type. > > The appropriate time to define that lowest-common-denominator > configuration format is *after* there is a working programmatic API > that covers at least the 3 major implementations of interest (OpenSSL, > SecureTransport, SChannel), and hopefully a few other implementations > as well (e.g. NSS, BoringSSL). > So, rather than scattered throughout each implementation, I think it would be wise to provide guidance regarding configuration validation. Enumerating the parameters into a common schema certainly will require actual implementations to define a superset of common dict keys; which I believe would be easier with a standard SSLConfig object. Another reason I believe there should be a configuration object with a .validate() method for centralized SSL configuration: - Having one .validate() method (with as many necessary subvalidators) provides a hook for security-conscious organizations to do SSL/TLS configuration validation in one place. (Otherwise, this type of configuration-validation must be frequently-re-implemented in an ad-hoc way; which inopportunely admits errors) - Examples of SSL/TLS configuration validation criteria: - https://en.wikipedia.org/wiki/FIPS_140-2#Cryptographic_Module_Validation_Program - https://mozilla.github.io/server-side-tls/ssl-config-generator/ - "CIS Critical Security Controls" https://www.cisecurity.org/critical-controls.cfm - CSC 3.[*] - Least-privilege dictates that this type of config is separate from the code.py files - "CWE-327: Use of a Broken or Risky Cryptographic Algorithm" (2011 Top 25 #19) https://cwe.mitre.org/top25/#CWE-327 https://cwe.mitre.org/data/definitions/327.html - ... "improper configuration" is basically this issue; so, validation should be encouraged where possible. Again, I think we should encourage validation of SSL/TLS configuration settings; and pointing to SSLConfig.validate() as the method to subclass makes that very easy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jan 13 03:04:29 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Jan 2017 18:04:29 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: On 13 January 2017 at 16:32, Wes Turner wrote: > Another reason I believe there should be a configuration object with a > .validate() method for centralized SSL configuration: > > - Having one .validate() method (with as many necessary subvalidators) > provides a hook for security-conscious organizations to do SSL/TLS > configuration validation in one place. (Otherwise, this type of > configuration-validation must be frequently-re-implemented in an ad-hoc way; > which inopportunely admits errors) Ah, I see what you mean you mean now - not "Is this a valid configuration for the current TLS implementation?", but rather "Does this TLS context, as configured, comply with a given security policy?" That also gets back to Christian's observation that: ========== We have to consider different kinds of min and max version: 1) min and max offical SSL/TLS standards 2) min and max supported standards for a TLS implementation 3) min and max version we consider as secure enough ========== Where points 1 & 2 are descriptive of the standards and particular implementations, point 3 is really about security policy recommendations and enforcement. I'd still be inclined to keep that concern separate from those of the core TLS context (which is about applying a given security context to sockets and buffers) though, and instead have the specific concept of a TLSPolicy that can be used to create TLS contexts that meet certain criteria: class TLSPolicy(abc.ABC): @abstractmethod def get_context(self, context_factory: Callable[[], TLSContext] = None) -> None: """Create a TLS context instance that complies with the policy""" ... @abstractmethod def validate_context(self, context: TLSContext) -> bool: """Returns True if the context complies with the policy, False otherwise""" ... The standard library could then provide a concrete implementation and default policy that was sufficient to enforce minimum TLS versions and the acceptable cipher suites, while still leaving room for more capable third party policy enforcement. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Fri Jan 13 03:27:30 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 13 Jan 2017 08:27:30 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: > On 13 Jan 2017, at 08:04, Nick Coghlan wrote: > > I'd still be inclined to keep that concern separate from those of the > core TLS context (which is about applying a given security context to > sockets and buffers) though, and instead have the specific concept of > a TLSPolicy that can be used to create TLS contexts that meet certain > criteria: ? snip ... > The standard library could then provide a concrete implementation and > default policy that was sufficient to enforce minimum TLS versions and > the acceptable cipher suites, while still leaving room for more > capable third party policy enforcement. So, I?d like to rein this discussion in a bit here. The idea of being able to validate the security and correctness of a collection of TLS settings is a valuable idea. It is also vastly beyond the scope of the problem I?m trying to solve here. I want to very strongly resist the desire that a lot of the community will have to attach additional complexity and responsibility to this module. While it?s very understandable to want to do that (after all, we don?t revisit our standard library TLS functionality often), it will have the effect of chaining a concrete block to this PEP?s ankle and then throwing it into the water. I want us to develop a culture around TLS functionality where we are willing to continually iterate on it, and where the community feels willing-and-able to provide extensions and new functionality incrementally, rather than all at once. Let?s not have the perfect be the enemy of the good here. I highly recommend that people who are interested in TLS policy workshop a separate PEP that discusses what we should build to resolve that issue. But we should not block solving problem A (how do we use something that isn?t OpenSSL with the standard library) on solving problem B (how do we define and enforce TLS policy). Cory From njs at pobox.com Fri Jan 13 03:48:15 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Jan 2017 00:48:15 -0800 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: On Fri, Jan 13, 2017 at 12:27 AM, Cory Benfield wrote: > >> On 13 Jan 2017, at 08:04, Nick Coghlan wrote: >> >> I'd still be inclined to keep that concern separate from those of the >> core TLS context (which is about applying a given security context to >> sockets and buffers) though, and instead have the specific concept of >> a TLSPolicy that can be used to create TLS contexts that meet certain >> criteria: > > ? snip ... > >> The standard library could then provide a concrete implementation and >> default policy that was sufficient to enforce minimum TLS versions and >> the acceptable cipher suites, while still leaving room for more >> capable third party policy enforcement. > > So, I?d like to rein this discussion in a bit here. > > The idea of being able to validate the security and correctness of a collection of TLS settings is a valuable idea. It is also vastly beyond the scope of the problem I?m trying to solve here. I want to very strongly resist the desire that a lot of the community will have to attach additional complexity and responsibility to this module. While it?s very understandable to want to do that (after all, we don?t revisit our standard library TLS functionality often), it will have the effect of chaining a concrete block to this PEP?s ankle and then throwing it into the water. > > I want us to develop a culture around TLS functionality where we are willing to continually iterate on it, and where the community feels willing-and-able to provide extensions and new functionality incrementally, rather than all at once. Let?s not have the perfect be the enemy of the good here. I highly recommend that people who are interested in TLS policy workshop a separate PEP that discusses what we should build to resolve that issue. But we should not block solving problem A (how do we use something that isn?t OpenSSL with the standard library) on solving problem B (how do we define and enforce TLS policy). The potentially-useful idea I took from this subthread was: right now we have an interface with a bunch of getter/setter methods. Would it make sense to *replace* that with something more declarative, like a single method that takes configuration data in some sort of standard format? Something very non-clever and simple, like, a dict? That seems like it might both simplify the interface (e.g. the sni callback can return one of these dicts instead of trying to specify OpenSSL's context switcheroo weirdness; if the new dict tries to change options that this particular implementation doesn't allow to be changed, then it's free to error out) and be more flexible (e.g. people could write validation routines that take one of these dicts and raise an error or not -- having a standard format enables this but also makes it not our problem). -n -- Nathaniel J. Smith -- https://vorpus.org From cory at lukasa.co.uk Fri Jan 13 03:55:24 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 13 Jan 2017 08:55:24 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> > On 13 Jan 2017, at 08:48, Nathaniel Smith wrote: > > The potentially-useful idea I took from this subthread was: right now > we have an interface with a bunch of getter/setter methods. Would it > make sense to *replace* that with something more declarative, like a > single method that takes configuration data in some sort of standard > format? Something very non-clever and simple, like, a dict? That seems > like it might both simplify the interface (e.g. the sni callback can > return one of these dicts instead of trying to specify OpenSSL's > context switcheroo weirdness; if the new dict tries to change options > that this particular implementation doesn't allow to be changed, then > it's free to error out) and be more flexible (e.g. people could write > validation routines that take one of these dicts and raise an error or > not -- having a standard format enables this but also makes it not our > problem). Hmm. I am extremely unsure. At the risk of sounding facetious, these two solutions are *basically* isomorphic to one another: that is, an object with a bunch of getters/setters is basically a dict, it?s just not something you can shove arbitrary data into. That?s my main reason for preferring an object to a dict: I want to eliminate a whole class of bugs that comes from accidentally doing `config[?ssl_version_minimum?] = TLSv1_1` instead of `config[?tls_version_minimum?] = TLSv1_1`. Put another way, the dict approach either requires that we override __setitem__ to validate all keys, or that we allow typo-based errors where people accidentally get the default. The second is terrifying, so we?d have to do the first, and at that point we?re basically just writing an object. ;) I think the actual key here is that the ?Context? object should not be confused with an *actual* Context object from the backing library. That is, a ClientContext() for OpenSSL does not need to compose onto it a SSL_CTX from OpenSSL. It is quite reasonable for the ClientContext to just be a bucket of data that manufactures SSL_CTX objects when wrap_socket is called. I don?t see why validation routines couldn?t run on Context objects directly. You could even do this: class ClientValidationContext(ClientContext): def validate(self): ???Checks internal state and returns whether or not the context meets your criteria??? pass Essentially, because all code operates on the abstract objects, one of the things you can do for validation is insert a *validating* implementation of the abstract object. It?s exactly like a test fake, but for validation. Does that make any sense, or have I not had enough coffee yet? Cory From cory at lukasa.co.uk Fri Jan 13 04:09:53 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 13 Jan 2017 09:09:53 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> Message-ID: <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> > On 13 Jan 2017, at 09:07, Nathaniel Smith wrote: > > I was imagining something along the lines of > > ClientContext({...}).wrap_socket(...) > > where ClientContext.__init__ would do the validation. Or there are > various other ways to slot the pieces together, but in general > validation doesn't need to be done incrementally; there's going to be > some place where the config dict hits the library and it can/should be > validated at that point. And before that point we get all the dict > niceties for free. What are the dict niceties, exactly? I?m still not seeing what the advantage is in throwing away a typed and restricted API for a dict is. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jan 13 04:07:42 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Jan 2017 01:07:42 -0800 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> Message-ID: On Fri, Jan 13, 2017 at 12:55 AM, Cory Benfield wrote: > >> On 13 Jan 2017, at 08:48, Nathaniel Smith wrote: >> >> The potentially-useful idea I took from this subthread was: right now >> we have an interface with a bunch of getter/setter methods. Would it >> make sense to *replace* that with something more declarative, like a >> single method that takes configuration data in some sort of standard >> format? Something very non-clever and simple, like, a dict? That seems >> like it might both simplify the interface (e.g. the sni callback can >> return one of these dicts instead of trying to specify OpenSSL's >> context switcheroo weirdness; if the new dict tries to change options >> that this particular implementation doesn't allow to be changed, then >> it's free to error out) and be more flexible (e.g. people could write >> validation routines that take one of these dicts and raise an error or >> not -- having a standard format enables this but also makes it not our >> problem). > > Hmm. > > I am extremely unsure. At the risk of sounding facetious, these two solutions are *basically* isomorphic to one another: that is, an object with a bunch of getters/setters is basically a dict, it?s just not something you can shove arbitrary data into. > > That?s my main reason for preferring an object to a dict: I want to eliminate a whole class of bugs that comes from accidentally doing `config[?ssl_version_minimum?] = TLSv1_1` instead of `config[?tls_version_minimum?] = TLSv1_1`. Put another way, the dict approach either requires that we override __setitem__ to validate all keys, or that we allow typo-based errors where people accidentally get the default. The second is terrifying, so we?d have to do the first, and at that point we?re basically just writing an object. ;) > > I think the actual key here is that the ?Context? object should not be confused with an *actual* Context object from the backing library. That is, a ClientContext() for OpenSSL does not need to compose onto it a SSL_CTX from OpenSSL. It is quite reasonable for the ClientContext to just be a bucket of data that manufactures SSL_CTX objects when wrap_socket is called. > > I don?t see why validation routines couldn?t run on Context objects directly. You could even do this: > > class ClientValidationContext(ClientContext): > def validate(self): > ???Checks internal state and returns whether or not the context meets your criteria??? > pass > > Essentially, because all code operates on the abstract objects, one of the things you can do for validation is insert a *validating* implementation of the abstract object. It?s exactly like a test fake, but for validation. > > Does that make any sense, or have I not had enough coffee yet? I was imagining something along the lines of ClientContext({...}).wrap_socket(...) where ClientContext.__init__ would do the validation. Or there are various other ways to slot the pieces together, but in general validation doesn't need to be done incrementally; there's going to be some place where the config dict hits the library and it can/should be validated at that point. And before that point we get all the dict niceties for free. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Fri Jan 13 04:33:52 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Jan 2017 01:33:52 -0800 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: On Fri, Jan 13, 2017 at 1:09 AM, Cory Benfield wrote: > > On 13 Jan 2017, at 09:07, Nathaniel Smith wrote: > > I was imagining something along the lines of > > ClientContext({...}).wrap_socket(...) > > where ClientContext.__init__ would do the validation. Or there are > various other ways to slot the pieces together, but in general > validation doesn't need to be done incrementally; there's going to be > some place where the config dict hits the library and it can/should be > validated at that point. And before that point we get all the dict > niceties for free. > > > What are the dict niceties, exactly? I?m still not seeing what the advantage > is in throwing away a typed and restricted API for a dict is. Getters & setters (right now the spec has setters for everything, but a getter for validate_certificates only -- that's kinda weird?), an obvious serialization story for the parts of the config that use basic types (e.g. to/from JSON/TOML/YAML), an obvious/natural way to represent "here are the config changes I want to make in response to that ClientHello" for the sni callback as a declarative object, a nice __repr__, familiarity, ... obviously these are all things one can put into the spec piece by piece, but you're the one who said you wanted to simplify things :-). I get what you're saying about a typed API -- basically in the current setup, the way _BaseContext is written as a bunch of Python methods means that the interpreter will automatically catch if someone tries to call set_cihpers, whereas in the dict version each implementation would have to catch this itself. But in practice this difference seems really minimal to me -- either way it's a runtime check, and it's really difficult to write a loop like for key, value in config.items(): do_some_quirky_cffi_thing_based_on_key() that doesn't implicitly validate the keys anyway. There's also the option of using e.g. JSON-Schema to write down a formal description of what goes in the dict -- this could actually be substantially stricter than what Python gets you, because you can actually statically enforce that validate_certificates is a bool. For whatever that's worth -- probably not much. If we really want to simplify it down, then as Alex noted in the PR, TLSWrappedSockets are superfluous, so we could make it just: # Simple functions, not methods: client_wrap_buffers(config, incoming, outgoing) -> TLSWrappedBuffers server_wrap_buffers(config, incoming, outgoing) -> TLSWrappedBuffers class TLSWrappedBuffers: ... read and all that stuff ... @property @abstractmethod def get_config(self) -> dict: pass def update_config(self, new_settings): # Attempt to do the equivalent of self.config.update(new_settings); # raises TLSError if any of the settings can't be changed given # current backend or current connection state. Anyway, I'm not actually sure if this is a good idea, but it seemed worth putting the two approaches up where we could look at them and compare :-). -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Fri Jan 13 06:02:26 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Jan 2017 21:02:26 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: On 13 January 2017 at 18:27, Cory Benfield wrote: > >> On 13 Jan 2017, at 08:04, Nick Coghlan wrote: >> >> I'd still be inclined to keep that concern separate from those of the >> core TLS context (which is about applying a given security context to >> sockets and buffers) though, and instead have the specific concept of >> a TLSPolicy that can be used to create TLS contexts that meet certain >> criteria: > > ? snip ... > >> The standard library could then provide a concrete implementation and >> default policy that was sufficient to enforce minimum TLS versions and >> the acceptable cipher suites, while still leaving room for more >> capable third party policy enforcement. > > So, I?d like to rein this discussion in a bit here. > > The idea of being able to validate the security and correctness of a collection of TLS settings is a valuable idea. It is also vastly beyond the scope of the problem I?m trying to solve here. I want to very strongly resist the desire that a lot of the community will have to attach additional complexity and responsibility to this module. While it?s very understandable to want to do that (after all, we don?t revisit our standard library TLS functionality often), it will have the effect of chaining a concrete block to this PEP?s ankle and then throwing it into the water. Note that I'd be reasonably happy to have my post interpreted as "We can leave policy definition and management until later without really hindering our ability to add it whenever we want to" - the only potential change to TLSContext would be to accept an optional "policy" parameter when creating instances, and that's relatively easy to add to implementations in a backwards compatible way (the only slight awkwardness lies in deciding whether or not to pass it along to base classes during initialization). However, I'm also aware that one of the awkwards points in the current SSL module is the significant difference in behaviour between calling ssl.create_default_context() (good defaults) and instantiating ssl.SSLContext directly (bad defaults). These are two different SSL/TLS security policies represented as distinct SSLContext factories. For the new module, that can be changed such that tls.TLSContext has inherently reasonable defaults and a priori notice that the defaults may change in maintenance releases, so we only have a single TLSContext factory and a single default TLS configutation policy. Unfortunately, that would still leave us with the question of how an API user can compare their own default settings to the standard library ones (e.g. to emit a warning or error when testing, even if they've opted out of relying directly on the defaults at runtime). If I'm understanding Wes correctly, this is the gist of the problem he's worried about as well - it doesn't do any of us or the overall ecosystem any good if incrementally evolving the default settings in the standard library leads to a significant number of developers instead defining their own "snapshot in time" TLS settings and then never updating them. By contrast, if the "default TLS policy" is pulled out as a distinct object that the tls.TLSContext constructor *uses* to configure the instance, then it lets us say "If you don't want to rely on the default settings at runtime, at least validate your own defaults against the standard library's TLS policy as part of your test suite". So overall, I think separating out "the standard library's default TLS settings" as an importable and introspectable object may actually make the overall design simpler and easier to explain. It just doesn't look that way right now because the initial draft glosses over the question of how to define and manage the default security policy (which is an entirely reasonable approach to have taken!). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at cheimes.de Fri Jan 13 06:17:47 2017 From: christian at cheimes.de (Christian Heimes) Date: Fri, 13 Jan 2017 12:17:47 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> Message-ID: <6024e06f-b4ab-6f8a-a135-ff7ad3cc53d6@cheimes.de> On 2017-01-13 12:02, Nick Coghlan wrote: > On 13 January 2017 at 18:27, Cory Benfield wrote: >> >>> On 13 Jan 2017, at 08:04, Nick Coghlan wrote: >>> >>> I'd still be inclined to keep that concern separate from those of the >>> core TLS context (which is about applying a given security context to >>> sockets and buffers) though, and instead have the specific concept of >>> a TLSPolicy that can be used to create TLS contexts that meet certain >>> criteria: >> >> ? snip ... >> >>> The standard library could then provide a concrete implementation and >>> default policy that was sufficient to enforce minimum TLS versions and >>> the acceptable cipher suites, while still leaving room for more >>> capable third party policy enforcement. >> >> So, I?d like to rein this discussion in a bit here. >> >> The idea of being able to validate the security and correctness of a collection of TLS settings is a valuable idea. It is also vastly beyond the scope of the problem I?m trying to solve here. I want to very strongly resist the desire that a lot of the community will have to attach additional complexity and responsibility to this module. While it?s very understandable to want to do that (after all, we don?t revisit our standard library TLS functionality often), it will have the effect of chaining a concrete block to this PEP?s ankle and then throwing it into the water. > > Note that I'd be reasonably happy to have my post interpreted as "We > can leave policy definition and management until later without really > hindering our ability to add it whenever we want to" - the only > potential change to TLSContext would be to accept an optional "policy" > parameter when creating instances, and that's relatively easy to add > to implementations in a backwards compatible way (the only slight > awkwardness lies in deciding whether or not to pass it along to base > classes during initialization). > > However, I'm also aware that one of the awkwards points in the current > SSL module is the significant difference in behaviour between calling > ssl.create_default_context() (good defaults) and instantiating > ssl.SSLContext directly (bad defaults). These are two different > SSL/TLS security policies represented as distinct SSLContext > factories. I'm addressing this issue with a new PEP (WIP) independently from Cory's PEP. In 3.6 SSLContext got a bit more sane. In the future we are going to have two protocols only (TLS_PROTOCOL_SERVER and TLS_PROTOCOL_CLIENT) with sane default values for both contexts. Client side context will default to cert and hostname verification, good cipher selection but no root CA certs. In fact I'm planning to write two PEPs, the first addresses the 'secure by default' issue and the second addresses various issues with hostname verification [1]. The second PEP will also require OpenSSL >= 1.0.2 and LibreSSL >= 2.5.0, because it depends on X509_VERIFY_PARAM_set1_host(). Christian [1] https://speakerdeck.com/tiran/pyconpl-2016-keynote-tales-from-python-security?slide=63 From ncoghlan at gmail.com Fri Jan 13 06:17:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 13 Jan 2017 21:17:56 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: On 13 January 2017 at 19:33, Nathaniel Smith wrote: > On Fri, Jan 13, 2017 at 1:09 AM, Cory Benfield wrote: > I get what you're saying about a typed API -- basically in the current > setup, the way _BaseContext is written as a bunch of Python methods > means that the interpreter will automatically catch if someone tries > to call set_cihpers, whereas in the dict version each implementation > would have to catch this itself. But in practice this difference seems > really minimal to me -- either way it's a runtime check The difference isn't minimal at all from the perspective of: - static type analysis - runtime introspection - IDEs (which use a mixture of both) - documentation Config dictionary based APIs have *major* problems in all those respects, as they're ultimately just a bunch of opaque-to-the-compiler keys and values. Network security is a sufficiently hard problem that we want to make it as easy as possible for developers to bring additional tools for ensuring code correctness to bear :) > and it's > really difficult to write a loop like > > for key, value in config.items(): > do_some_quirky_cffi_thing_based_on_key() > > that doesn't implicitly validate the keys anyway. There's also the > option of using e.g. JSON-Schema to write down a formal description of > what goes in the dict -- this could actually be substantially stricter > than what Python gets you, because you can actually statically enforce > that validate_certificates is a bool. For whatever that's worth -- > probably not much. I've generally found that it's easier to build a declarative API atop a programmatic API than it is to tackle a problem the other way around: 1. Even if the only public API is declarative, there's going to be a programmatic API that implements the actual processing of the declarative requests 2. Working with the programmatic API provides insight into which features the declarative API should cover, and which it can skip 3. Programmatic APIs are often easier to test, since you can more readily isolate the effects of the individual operations 4. A public programmatic API serves as a natural constraint and test case for any subsequent declarative API Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Fri Jan 13 08:09:49 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 13 Jan 2017 05:09:49 -0800 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: On Jan 13, 2017 3:17 AM, "Nick Coghlan" wrote: On 13 January 2017 at 19:33, Nathaniel Smith wrote: > On Fri, Jan 13, 2017 at 1:09 AM, Cory Benfield wrote: > I get what you're saying about a typed API -- basically in the current > setup, the way _BaseContext is written as a bunch of Python methods > means that the interpreter will automatically catch if someone tries > to call set_cihpers, whereas in the dict version each implementation > would have to catch this itself. But in practice this difference seems > really minimal to me -- either way it's a runtime check The difference isn't minimal at all from the perspective of: - static type analysis - runtime introspection - IDEs (which use a mixture of both) - documentation Config dictionary based APIs have *major* problems in all those respects, as they're ultimately just a bunch of opaque-to-the-compiler keys and values. Network security is a sufficiently hard problem that we want to make it as easy as possible for developers to bring additional tools for ensuring code correctness to bear :) > and it's > really difficult to write a loop like > > for key, value in config.items(): > do_some_quirky_cffi_thing_based_on_key() > > that doesn't implicitly validate the keys anyway. There's also the > option of using e.g. JSON-Schema to write down a formal description of > what goes in the dict -- this could actually be substantially stricter > than what Python gets you, because you can actually statically enforce > that validate_certificates is a bool. For whatever that's worth -- > probably not much. I've generally found that it's easier to build a declarative API atop a programmatic API than it is to tackle a problem the other way around: 1. Even if the only public API is declarative, there's going to be a programmatic API that implements the actual processing of the declarative requests 2. Working with the programmatic API provides insight into which features the declarative API should cover, and which it can skip 3. Programmatic APIs are often easier to test, since you can more readily isolate the effects of the individual operations 4. A public programmatic API serves as a natural constraint and test case for any subsequent declarative API I totally buy these as advantages of a generic programmatic API over a generic declarative API, but I'm not sure how much it applies given that in this case the "programmatic" API we're talking about is literally *nothing* but getters and setters. Spelling set_ciphers correctly is certainly important and it's nice if your IDE can help, no doubt. But let's keep this in perspective - I don't think this is the *hard* part of network security, really :-). And I can't resist pointing out that in your other email sent at the same time, you're musing about out how nice it would be if only there were some way the stdlib could export its idea of a "default tls configuration" as some sort of concrete object that arbitrary tls implementations could ingest... which is *exactly* what the config-dict approach provides :-). I guess another possible point in the design space would be to split the difference: instead of an abstract context class or a dict, define a concrete context class that acts as a simple transparent container for these settings, effectively layering the type safety etc over its __dict__. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Fri Jan 13 08:56:04 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 13 Jan 2017 13:56:04 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: > On 13 Jan 2017, at 13:09, Nathaniel Smith wrote: > > I guess another possible point in the design space would be to split the difference: instead of an abstract context class or a dict, define a concrete context class that acts as a simple transparent container for these settings, effectively layering the type safety etc over its __dict__. I think I?d want to do it this way if we were going to do it all: I?m extremely reluctant to use dicts and strings for this if we can possibly avoid it. In fact, you?ll note that the API goes to substantial lengths to avoid passing strings around in almost all cases, except where such a use is required essentially by convention (e.g. paths). While we?re here, I should point out that this does not replace the abstract contexts entirely, because we still need the wrap_* methods. These would now just be fed a Configuration object (I?m more comfortable with the name Configuration than Policy for this usage) in their constructor, and then could use the wrap_* methods as needed. How does that idea sound to the rest of the list? Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Jan 13 10:45:02 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Jan 2017 01:45:02 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: On 13 January 2017 at 23:56, Cory Benfield wrote: > While we?re here, I should point out that this does not replace the abstract > contexts entirely, because we still need the wrap_* methods. These would now > just be fed a Configuration object (I?m more comfortable with the name > Configuration than Policy for this usage) in their constructor, and then > could use the wrap_* methods as needed. Having a relatively passive configuration object sounds OK to me, and I agree it's distinct from a Policy object (which would store constraints on what an acceptable configuration looks like, rather than a specific configuration). > How does that idea sound to the rest of the list? It would definitely address my concern about making it easy for people to re-use the standard library's default TLS configuration settings, and it would also make it easier to have different defaults for different purposes. So the essential stack would look like: TLSConfig[uration?]: implementation independent, settings only TLSClientContext: ABC to combine settings with a specific TLS implementation TLSServerContext: ABC to combine settings with a specific TLS implementation TLSSocket: ABC to combine a context with a network socket TLSBuffer: ABC to combine a context with a pair of data buffers And then TLSPolicy would be a potential future implementation independent addition that could be used to constrain acceptable TLS configurations. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Fri Jan 13 10:58:01 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 13 Jan 2017 15:58:01 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: > On 13 Jan 2017, at 15:45, Nick Coghlan wrote: > > So the essential stack would look like: > > TLSConfig[uration?]: implementation independent, settings only > TLSClientContext: ABC to combine settings with a specific TLS implementation > TLSServerContext: ABC to combine settings with a specific TLS implementation > TLSSocket: ABC to combine a context with a network socket > TLSBuffer: ABC to combine a context with a pair of data buffers > > And then TLSPolicy would be a potential future implementation > independent addition that could be used to constrain acceptable TLS > configurations. If we were going this way, I?d want to add one extra caveat: I think I?d want the Contexts to become immutable. The logic for the SNI callback would then become: you are called with the Context that created the socket/buffer, and you return a Configuration object that contains any changes you want to make, and the Context applies them if it can (or errors out if it cannot). A new Context is created. This relegates Context to the role of ?socket/buffer factory?. The advantage of this is that we have vastly reduced the moving parts: a Context can ensure that, once initiated, the Policy that belongs to it will not change under its feet. It also allows the Context to refuse to change settings that a given concrete implementation cannot change in the SNI callback. Essentially, the logic in the callback would be: def sni_callback(buffer, hostname, context): # This creates a writable copy of the configuration: it does not # mutate the original. configuration = context.configuration configuration.certificates = certs_for_hostname(hostname) configuration.inner_protocols = [NextProtocol.H2, NextProtocol.HTTP1] return configuration This would almost certainly make Context implementation easier, as there is no longer a requirement to monitor your configuration and support live-updates. Cory From ncoghlan at gmail.com Fri Jan 13 11:05:55 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Jan 2017 02:05:55 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <99d72b75-14bf-0d05-c8e1-e49f2b0a88fb@cheimes.de> <5f0720e9-5f33-b7f7-4fdf-30d977822e98@cheimes.de> <4964BA95-0819-4C84-A660-BD836C8CC4F4@stufft.io> <8F13F785-B466-468A-A605-6F8729713875@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: On 14 January 2017 at 01:58, Cory Benfield wrote: > >> On 13 Jan 2017, at 15:45, Nick Coghlan wrote: >> >> So the essential stack would look like: >> >> TLSConfig[uration?]: implementation independent, settings only >> TLSClientContext: ABC to combine settings with a specific TLS implementation >> TLSServerContext: ABC to combine settings with a specific TLS implementation >> TLSSocket: ABC to combine a context with a network socket >> TLSBuffer: ABC to combine a context with a pair of data buffers >> >> And then TLSPolicy would be a potential future implementation >> independent addition that could be used to constrain acceptable TLS >> configurations. > > If we were going this way, I?d want to add one extra caveat: I think I?d want the Contexts to become immutable. > > The logic for the SNI callback would then become: you are called with the Context that created the socket/buffer, and you return a Configuration object that contains any changes you want to make, and the Context applies them if it can (or errors out if it cannot). A new Context is created. This relegates Context to the role of ?socket/buffer factory?. The advantage of this is that we have vastly reduced the moving parts: a Context can ensure that, once initiated, the Policy that belongs to it will not change under its feet. It also allows the Context to refuse to change settings that a given concrete implementation cannot change in the SNI callback. Having immutable-by-default config with explicitly managed state changes in particular well-defined scenarios sounds like a big win to me. > This would almost certainly make Context implementation easier, as there is no longer a requirement to monitor your configuration and support live-updates. It also drastically reduces the test matrix required, as eliminating any potential dependence on the order in which settings are applied, means you only need to test interesting *combinations* of settings, and not different configuration *sequences*. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at cheimes.de Fri Jan 13 11:20:46 2017 From: christian at cheimes.de (Christian Heimes) Date: Fri, 13 Jan 2017 17:20:46 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: On 2017-01-13 16:45, Nick Coghlan wrote: > On 13 January 2017 at 23:56, Cory Benfield wrote: >> While we?re here, I should point out that this does not replace the abstract >> contexts entirely, because we still need the wrap_* methods. These would now >> just be fed a Configuration object (I?m more comfortable with the name >> Configuration than Policy for this usage) in their constructor, and then >> could use the wrap_* methods as needed. > > Having a relatively passive configuration object sounds OK to me, and > I agree it's distinct from a Policy object (which would store > constraints on what an acceptable configuration looks like, rather > than a specific configuration). > >> How does that idea sound to the rest of the list? > > It would definitely address my concern about making it easy for people > to re-use the standard library's default TLS configuration settings, > and it would also make it easier to have different defaults for > different purposes. > > So the essential stack would look like: > > TLSConfig[uration?]: implementation independent, settings only > TLSClientContext: ABC to combine settings with a specific TLS implementation > TLSServerContext: ABC to combine settings with a specific TLS implementation > TLSSocket: ABC to combine a context with a network socket > TLSBuffer: ABC to combine a context with a pair of data buffers > > And then TLSPolicy would be a potential future implementation > independent addition that could be used to constrain acceptable TLS > configurations. There aren't many settings that are truly implementation independent. Even ciphers depend on the implementation and version of the TLS provider. Some implementations do support more or less ciphers. Some allow ordering or black listing of algorithms, some do not. Apparently it's even hard to not use the system trust store in some implementations (SecureTransport). How should we deal with e.g. when a TLS implementation does not accept or now about a cipher? I would rather invest time to make TLSConfiguration optional for 99% of all users. Unless you need to connect to a crappy legacy device, a user or developer should not need to mess with TLS settings to get secure options. Some TLS libraries already set sane defaults. OpenSSL is getting there, too. Browsers get it right as well. Even for OpenSSL 1.0.2 (first 1.0.2 release) it is possible to set a secure and future proof cipher list: [openssl/1.0.2]$ bin/openssl ciphers 'DEFAULT:!3DES:!EXPORT:!RC4:!DES:!MD5:!SRP:!PSK:!IDEA:!SEED' | sed 's/:/\n/g' ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES256-SHA ECDHE-ECDSA-AES256-SHA DH-DSS-AES256-GCM-SHA384 DHE-DSS-AES256-GCM-SHA384 DH-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES256-SHA256 DHE-DSS-AES256-SHA256 DH-RSA-AES256-SHA256 DH-DSS-AES256-SHA256 DHE-RSA-AES256-SHA DHE-DSS-AES256-SHA DH-RSA-AES256-SHA DH-DSS-AES256-SHA DHE-RSA-CAMELLIA256-SHA DHE-DSS-CAMELLIA256-SHA DH-RSA-CAMELLIA256-SHA DH-DSS-CAMELLIA256-SHA ECDH-RSA-AES256-GCM-SHA384 ECDH-ECDSA-AES256-GCM-SHA384 ECDH-RSA-AES256-SHA384 ECDH-ECDSA-AES256-SHA384 ECDH-RSA-AES256-SHA ECDH-ECDSA-AES256-SHA AES256-GCM-SHA384 AES256-SHA256 AES256-SHA CAMELLIA256-SHA ECDHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES128-SHA ECDHE-ECDSA-AES128-SHA DH-DSS-AES128-GCM-SHA256 DHE-DSS-AES128-GCM-SHA256 DH-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA256 DHE-DSS-AES128-SHA256 DH-RSA-AES128-SHA256 DH-DSS-AES128-SHA256 DHE-RSA-AES128-SHA DHE-DSS-AES128-SHA DH-RSA-AES128-SHA DH-DSS-AES128-SHA DHE-RSA-CAMELLIA128-SHA DHE-DSS-CAMELLIA128-SHA DH-RSA-CAMELLIA128-SHA DH-DSS-CAMELLIA128-SHA ECDH-RSA-AES128-GCM-SHA256 ECDH-ECDSA-AES128-GCM-SHA256 ECDH-RSA-AES128-SHA256 ECDH-ECDSA-AES128-SHA256 ECDH-RSA-AES128-SHA ECDH-ECDSA-AES128-SHA AES128-GCM-SHA256 AES128-SHA256 AES128-SHA CAMELLIA128-SHA From christian at cheimes.de Fri Jan 13 11:35:13 2017 From: christian at cheimes.de (Christian Heimes) Date: Fri, 13 Jan 2017 17:35:13 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: <20262400-5963-5363-70ce-441bd738d2d7@cheimes.de> On 2017-01-13 16:58, Cory Benfield wrote: > >> On 13 Jan 2017, at 15:45, Nick Coghlan wrote: >> >> So the essential stack would look like: >> >> TLSConfig[uration?]: implementation independent, settings only >> TLSClientContext: ABC to combine settings with a specific TLS implementation >> TLSServerContext: ABC to combine settings with a specific TLS implementation >> TLSSocket: ABC to combine a context with a network socket >> TLSBuffer: ABC to combine a context with a pair of data buffers >> >> And then TLSPolicy would be a potential future implementation >> independent addition that could be used to constrain acceptable TLS >> configurations. > > If we were going this way, I?d want to add one extra caveat: I think I?d want the Contexts to become immutable. > > The logic for the SNI callback would then become: you are called with the Context that created the socket/buffer, and you return a Configuration object that contains any changes you want to make, and the Context applies them if it can (or errors out if it cannot). A new Context is created. This relegates Context to the role of ?socket/buffer factory?. The advantage of this is that we have vastly reduced the moving parts: a Context can ensure that, once initiated, the Policy that belongs to it will not change under its feet. It also allows the Context to refuse to change settings that a given concrete implementation cannot change in the SNI callback. > > Essentially, the logic in the callback would be: > > def sni_callback(buffer, hostname, context): > # This creates a writable copy of the configuration: it does not > # mutate the original. > configuration = context.configuration > configuration.certificates = certs_for_hostname(hostname) > configuration.inner_protocols = [NextProtocol.H2, NextProtocol.HTTP1] > return configuration > > This would almost certainly make Context implementation easier, as there is no longer a requirement to monitor your configuration and support live-updates. How would this work for OpenSSL? In OpenSSL the SNI callback replaces the SSL_CTX of a SSL socket pointer with another SSL_CTX. The new SSL_CTX takes care of cipher negotiation, certs and other handshake details. The SSL_CTX should be reused in order to benefit from cached certs, HSM stuff and cached sessions. OpenSSL binds sessions to SSL_CTX instances. A callback looks more like this: contexts = { 'www.example.org': SSLContext(cert1, key1), 'internal.example.com': SSLContext(cert2, key2), } def sni_callback(sock, hostname): sock.context = contexts[hostname] Christian From cory at lukasa.co.uk Fri Jan 13 11:37:14 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 13 Jan 2017 16:37:14 +0000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <20262400-5963-5363-70ce-441bd738d2d7@cheimes.de> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> <20262400-5963-5363-70ce-441bd738d2d7@cheimes.de> Message-ID: <65E3CA5B-A241-478A-9E62-2BB5CE519A24@lukasa.co.uk> > On 13 Jan 2017, at 16:35, Christian Heimes wrote: > > How would this work for OpenSSL? In OpenSSL the SNI callback replaces > the SSL_CTX of a SSL socket pointer with another SSL_CTX. The new > SSL_CTX takes care of cipher negotiation, certs and other handshake > details. The SSL_CTX should be reused in order to benefit from cached > certs, HSM stuff and cached sessions. OpenSSL binds sessions to SSL_CTX > instances. > > A callback looks more like this: > > contexts = { > 'www.example.org': SSLContext(cert1, key1), > 'internal.example.com': SSLContext(cert2, key2), > } > > def sni_callback(sock, hostname): > sock.context = contexts[hostname] If the goal is to keep those contexts static, the best thing to do is to cache the context based on the configuration. Because configurations should be static they should be hashable, which would mean that the ServerContext can keep an internal dictionary of {configuration: SSLContext}. When the new configuration is returned, it can simply pull the context out of the cache as needed. Cory From ncoghlan at gmail.com Fri Jan 13 11:37:23 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Jan 2017 02:37:23 +1000 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> Message-ID: On 14 January 2017 at 02:20, Christian Heimes wrote: > There aren't many settings that are truly implementation independent. > Even ciphers depend on the implementation and version of the TLS > provider. Some implementations do support more or less ciphers. Some > allow ordering or black listing of algorithms, some do not. Apparently > it's even hard to not use the system trust store in some implementations > (SecureTransport). How should we deal with e.g. when a TLS > implementation does not accept or now about a cipher? That problem is one we're going to have to resolve regardless, and could potentially become its own configuration setting. > I would rather invest time to make TLSConfiguration optional for 99% of > all users. Unless you need to connect to a crappy legacy device, a user > or developer should not need to mess with TLS settings to get secure > options. Some TLS libraries already set sane defaults. OpenSSL is > getting there, too. Browsers get it right as well. Sure, but there are also plenty of folks that *are* going to need to tinker with those settings: - stdlib maintainers - folks writing TLS servers - folks writing TLS client libraries - folks writing test cases for TLS servers - folks writing test cases for TLS clients - penetration testers - network security researchers - folks integrating with legacy infrastructure - embedded software developers - etc We want to provide those folks with a reasonable learning curve as they move away from "the defaults are good enough for me" to "I need to make these changes, for these reasons", even as we attempt to make sure that "routine" use cases are secure by default. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at cheimes.de Fri Jan 13 11:46:54 2017 From: christian at cheimes.de (Christian Heimes) Date: Fri, 13 Jan 2017 17:46:54 +0100 Subject: [Security-sig] Unified TLS API for Python In-Reply-To: <65E3CA5B-A241-478A-9E62-2BB5CE519A24@lukasa.co.uk> References: <0A96806A-94A6-4D2C-8E8A-EF46612BDF82@lukasa.co.uk> <0E10CEC6-9A17-4D18-AED2-DA3BA0476150@lukasa.co.uk> <46AD5F15-429B-4589-9F44-0DD866449A0D@lukasa.co.uk> <20262400-5963-5363-70ce-441bd738d2d7@cheimes.de> <65E3CA5B-A241-478A-9E62-2BB5CE519A24@lukasa.co.uk> Message-ID: <56d5f8e2-b7fb-0c5f-3ca0-460335e79f3b@cheimes.de> On 2017-01-13 17:37, Cory Benfield wrote: > >> On 13 Jan 2017, at 16:35, Christian Heimes wrote: >> >> How would this work for OpenSSL? In OpenSSL the SNI callback replaces >> the SSL_CTX of a SSL socket pointer with another SSL_CTX. The new >> SSL_CTX takes care of cipher negotiation, certs and other handshake >> details. The SSL_CTX should be reused in order to benefit from cached >> certs, HSM stuff and cached sessions. OpenSSL binds sessions to SSL_CTX >> instances. >> >> A callback looks more like this: >> >> contexts = { >> 'www.example.org': SSLContext(cert1, key1), >> 'internal.example.com': SSLContext(cert2, key2), >> } >> >> def sni_callback(sock, hostname): >> sock.context = contexts[hostname] > > If the goal is to keep those contexts static, the best thing to do is to cache the context based on the configuration. Because configurations should be static they should be hashable, which would mean that the ServerContext can keep an internal dictionary of {configuration: SSLContext}. When the new configuration is returned, it can simply pull the context out of the cache as needed. You lost me and I'm tired. My brain is no longer able to follow. I'm calling it a day. A working example or PoC might help to understand your point... :) Christian From cory at lukasa.co.uk Thu Jan 19 12:29:23 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 19 Jan 2017 17:29:23 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 Message-ID: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> All, Thanks for your feedback on the draft PEP I proposed last week! There was a lot of really enthusiastic and valuable feedback both on this mailing list and on GitHub. I believe I?ve addressed a lot of the concerns that were brought up with the PEP now, so I?d like to ask that interested parties take another look. Some quick hits on the changes: - I?ve introduced a TLSConfiguration object and extracted all configuration from the Context onto it. This object is an immutable one (it?s strictly a slightly-fancy namedtuple), which provides some really huge advantages in terms of managing them and allowing concrete implementations to use Configuration objects as dictionary keys. - I?ve fleshed out the cipher suite section. - I?ve split out read_into. - I?ve stopped allowing passwords to be strings: they must now always be bytes. - I?ve dramatically changed the SNI callback to take advantage of the TLSConfiguration. - I?ve added support for trust stores. - Several other smaller changes. Please let me know what you think. The version of the draft PEP, from commit ce74bc60, is reproduced below. Thanks! Cory ? PEP: XXX Title: TLS Abstract Base Classes Version: $Revision$ Last-Modified: $Date$ Author: Cory Benfield Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 17-Oct-2016 Python-Version: 3.7 Post-History: 30-Aug-2002 Abstract ======== This PEP would define a standard TLS interface in the form of a collection of abstract base classes. This interface would allow Python implementations and third-party libraries to provide bindings to TLS libraries other than OpenSSL that can be used by tools that expect the interface provided by the Python standard library, with the goal of reducing the dependence of the Python ecosystem on OpenSSL. Rationale ========= In the 21st century it has become increasingly clear that robust and user-friendly TLS support is an extremely important part of the ecosystem of any popular programming language. For most of its lifetime, this role in the Python ecosystem has primarily been served by the `ssl module`_, which provides a Python API to the `OpenSSL library`_. Because the ``ssl`` module is distributed with the Python standard library, it has become the overwhelmingly most-popular method for handling TLS in Python. An extraordinary majority of Python libraries, both in the standard library and on the Python Package Index, rely on the ``ssl`` module for their TLS connectivity. Unfortunately, the preeminence of the ``ssl`` module has had a number of unforseen side-effects that have had the effect of tying the entire Python ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even in situations where it may provide a worse user experience than alternative TLS implementations, which imposes a cognitive burden and makes it hard to provide "platform-native" experiences. Problems -------- The fact that the ``ssl`` module is build into the standard library has meant that all standard-library Python networking libraries are entirely reliant on the OpenSSL that the Python implementation has been linked against. This leads to the following issues: * It is difficult to take advantage of new, higher-security TLS without recompiling Python to get a new OpenSSL. While there are third-party bindings to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format that the standard library understands, forcing projects that want to use them to maintain substantial compatibility layers. * For Windows distributions of Python, they need to be shipped with a copy of OpenSSL. This puts the CPython development team in the position of being OpenSSL redistributors, potentially needing to ship security updates to the Windows Python distributions when OpenSSL vulnerabilities are released. * For macOS distributions of Python, they need either to be shipped with a copy of OpenSSL or linked against the system OpenSSL library. Apple has formally deprecated linking against the system OpenSSL library, and even if they had not, that library version has been unsupported by upstream for nearly one year as of the time of writing. The CPython development team has started shipping newer OpenSSLs with the Python available from python.org, but this has the same problem as with Windows. * Many systems, including but not limited to Windows and macOS, do not make their system certificate stores available to OpenSSL. This forces users to either obtain their trust roots from elsewhere (e.g. `certifi`_) or to attempt to export their system trust stores in some form. Relying on `certifi`_ is less than ideal, as most system administrators do not expect to receive security-critical software updates from PyPI. Additionally, it is not easy to extend the `certifi`_ trust bundle to include custom roots, or to centrally manage trust using the `certifi`_ model. Even in situations where the system certificate stores are made available to OpenSSL in some form, the experience is still sub-standard, as OpenSSL will perform different validation checks than the platform-native TLS implementation. This can lead to users experiencing different behaviour on their browsers or other platform-native tools than they experience in Python, with little or no recourse to resolve the problem. * Users may wish to integrate with TLS libraries other than OpenSSL for many other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), or because OpenSSL is simply too large and unweildy for the platform (e.g. for embedded Python). Those users are left with the requirement to use third-party networking libraries that can interact with their preferred TLS library or to shim their preferred library into the OpenSSL-specific ``ssl`` module API. Additionally, the ``ssl`` module as implemented today limits the ability of CPython itself to add support for alternative TLS backends, or remove OpenSSL support entirely, should either of these become necessary or useful. The ``ssl`` module exposes too many OpenSSL-specific function calls and features to easily map to an alternative TLS backend. Proposal ======== This PEP proposes to introduce a few new Abstract Base Classes in Python 3.7 to provide TLS functionality that is not so strongly tied to OpenSSL. It also proposes to update standard library modules to use only the interface exposed by these abstract base classes wherever possible. There are three goals here: 1. To provide a common API surface for both core and third-party developers to target their TLS implementations to. This allows TLS developers to provide interfaces that can be used by most Python code, and allows network developers to have an interface that they can target that will work with a wide range of TLS implementations. 2. To provide an API that has few or no OpenSSL-specific concepts leak through. The ``ssl`` module today has a number of warts caused by leaking OpenSSL concepts through to the API: the new ABCs would remove those specific concepts. 3. To provide a path for the core development team to make OpenSSL one of many possible TLS backends, rather than requiring that it be present on a system in order for Python to have TLS support. The proposed interface is laid out below. Abstract Base Classes --------------------- There are several interfaces that require standardisation. Those interfaces are: 1. Configuring TLS, currently implemented by the `SSLContext`_ class in the ``ssl`` module. 2. Wrapping a socket object, currently implemented by the `SSLSocket`_ class in the ``ssl`` module. 3. Providing an in-memory buffer for doing in-memory encryption or decryption with no actual I/O (necessary for asynchronous I/O models), currently implemented by the `SSLObject`_ class in the ``ssl`` module. 4. Applying TLS configuration to the wrapping objects in (2) and (3). Currently this is also implemented by the `SSLContext`_ class in the ``ssl`` module. 5. Specifying TLS cipher suites. There is currently no code for doing this in the standard library: instead, the standard library uses OpenSSL cipher suite strings. 6. Specifying application-layer protocols that can be negotiated during the TLS handshake. 7. Specifying TLS versions. 8. Reporting errors to the caller, currently implemented by the `SSLError`_ class in the ``ssl`` module. 9. Specifying certificates to load, either as client or server certificates. 10. Specifying which trust database should be used to validate certificates presented by a remote peer. While it is technically possible to define (2) in terms of (3), for the sake of simplicity it is easier to define these as two separate ABCs. Implementations are of course free to implement the concrete subclasses however they see fit. Obviously, (5) doesn't require an abstract base class: instead, it requires a richer API for configuring supported cipher suites that can be easily updated with supported cipher suites for different implementations. (9) is a thorny problem, becuase in an ideal world the private keys associated with these certificates would never end up in-memory in the Python process (that is, the TLS library would collaborate with a Hardware Security Module (HSM) to provide the private key in such a way that it cannot be extracted from process memory). Thus, we need to provide an extensible model of providing certificates that allows concrete implementations the ability to provide this higher level of security, while also allowing a lower bar for those implementations that cannot. This lower bar would be the same as the status quo: that is, the certificate may be loaded from an in-memory buffer or from a file on disk. (10) also represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. Some implementations have specific trust store formats that only they can use (such as the OpenSSL CA directory format that is created by ``c_rehash``), and others may not allow you to specify a trust store that does not include their default trust store. For this reason, we need to provide a model that assumes very little about the form that trust stores take. The "Trust Store" section below goes into more detail about how this is achieved. Finally, this API will split the responsibilities currently assumed by the `SSLContext`_ object: specifically, the responsibility for holding and managing configuration and the responsibility for using that configuration to build wrapper objects. This is necessarily primarily for supporting functionality like Server Name Indication (SNI). In OpenSSL (and thus in the ``ssl`` module), the server has the ability to modify the TLS configuration in response to the client telling the server what hostname it is trying to reach. This is mostly used to change certificate chain so as to present the correct TLS certificate chain for the given hostname. The specific mechanism by which this is done is by returning a new `SSLContext`_ object with the appropriate configuration. This is not a model that maps well to other TLS implementations. Instead, we need to make it possible to provide a return value from the SNI callback that can be used to indicate what configuration changes should be made. This means providing an object that can hold TLS configuration. This object needs to be applied to specific TLSWrappedBuffer, and TLSWrappedSocket objects. For this reason, we split the responsibility of `SSLContext`_ into two separate objects. The ``TLSConfiguration`` object is an object that acts as container for TLS configuration: the ``ClientContext`` and ``ServerContext`` objects are objects that are instantiated with a ``TLSConfiguration`` object. Both objects would be immutable. Configuration ~~~~~~~~~~~~~ The ``TLSConfiguration`` concrete class defines an object that can hold and manage TLS configuration. The goals of this class are as follows: 1. To provide a method of specifying TLS configuration that avoids the risk of errors in typing (this excludes the use of a simple dictionary). 2. To provide an object that can be safely compared to other configuration objects to detect changes in TLS configuration, for use with the SNI callback. This class is not an ABC, primarily because it is not expected to have implementation-specific behaviour. The responsibility for transforming a ``TLSConfiguration`` object into a useful set of configuration for a given TLS implementation belongs to the Context objects discussed below. This class has one other notable property: it is immutable. This is a desirable trait for a few reasons. The most important one is that it allows these objects to be used as dictionary keys, which is potentially extremely valuable for certain TLS backends and their SNI configuration. On top of this, it frees implementations from needing to worry about their configuration objects being changed under their feet, which allows them to avoid needing to carefully synchronize changes between their concrete data structures and the configuration object. The ``TLSConfiguration`` object would be defined by the following code: ServerNameCallback = Callable[[TLSBufferObject, Optional[str], TLSConfiguration], Any] _configuration_fields = [ 'validate_certificates', 'certificate_chain', 'ciphers', 'inner_protocols', 'lowest_supported_version', 'highest_supported_version', 'trust_store', 'sni_callback', ] _DEFAULT_VALUE = object() class TLSConfiguration(namedtuple('TLSConfiguration', _configuration_fields)): """ An imutable TLS Configuration object. This object has the following properties: :param validate_certificates bool: Whether to validate the TLS certificates. This switch operates at a very broad scope: either validation is enabled, in which case all forms of validation are performed including hostname validation if possible, or validation is disabled, in which case no validation is performed. Not all backends support having their certificate validation disabled. If a backend does not support having their certificate validation disabled, attempting to set this property to ``False`` will throw a ``TLSError`` when this object is passed into a context object. :param certificate_chain Tuple[Tuple[Certificate],PrivateKey]: The certificate, intermediate certificate, and the corresponding private key for the leaf certificate. These certificates will be offered to the remote peer during the handshake if required. The first Certificate in the list must be the leaf certificate. All subsequent certificates will be offered as intermediate additional certificates. :param ciphers Tuple[CipherSuite]: The available ciphers for TLS connections created with this configuration, in priority order. :param inner_protocols Tuple[Union[NextProtocol, bytes]]: Protocols that connections created with this configuration should advertise as supported during the TLS handshake. These may be advertised using either or both of ALPN or NPN. This list of protocols should be ordered by preference. :param lowest_supported_version TLSVersion: The minimum version of TLS that should be allowed on TLS connections using this configuration. :param highest_supported_version TLSVersion: The maximum version of TLS that should be allowed on TLS connections using this configuration. :param trust_store TrustStore: The trust store that connections using this configuration will use to validate certificates. :param sni_callback Optional[ServerNameCallback]: A callback function that will be called after the TLS Client Hello handshake message has been received by the TLS server when the TLS client specifies a server name indication. Only one callback can be set per ``TLSConfiguration``. If the ``sni_callback`` is ``None`` then the callback is disabled. If the ``TLSConfiguration`` is used for a ``ClientContext`` then this setting will be ignored. The ``callback`` function will be called with three arguments: the first will be the ``TLSBufferObject`` for the connection; the second will be a string that represents the server name that the client is intending to communicate (or ``None`` if the TLS Client Hello does not contain a server name); and the third argument will be the original ``Context``. The server name argument will be the IDNA *decoded* server name. The ``callback`` must return a ``TLSConfiguration`` to allow negotiation to continue. Other return values signal errors. Attempting to control what error is signaled by the underlying TLS implementation is not specified in this API, but is up to the concrete implementation to handle. The Context will do its best to apply the ``TLSConfiguration`` changes from its original configuration to the incoming connection. This will usually include changing the certificate chain, but may also include changes to allowable ciphers or any other configuration settings. """ __slots__ = () def __new__(cls, validate_certificates=None: Optional[bool], certificate_chain=None: Optional[Tuple[Tuple[Certificate], PrivateKey]], ciphers=None: Optional[Tuple[CipherSuite]], inner_protocols=None: Optional[Tuple[Union[NextProtocol, bytes]]], lowest_supported_version=None: Optional[TLSVersion], highest_supported_version=None: Optional[TLSVersion], trust_store=None: Optional[TrustStore], sni_callback=None: Optional[ServerNameCallback]): if validate_certificates is None: validate_certificates = True if ciphers is None: ciphers = DEFAULT_CIPHER_LIST if inner_protocols is None: inner_protocols = [] if lowest_supported_version is None: lowest_supported_version = TLSVersion.TLSv1 if highest_supported_version is None: highest_supported_version = TLSVersion.MAXIMUM_SUPPORTED return super().__new__( cls, validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) def update(self, validate_certificates=_DEFAULT_VALUE, certificate_chain=_DEFAULT_VALUE, ciphers=_DEFAULT_VALUE, inner_protocols=_DEFAULT_VALUE, lowest_supported_version=_DEFAULT_VALUE, highest_supported_version=_DEFAULT_VALUE, trust_store=_DEFAULT_VALUE, sni_callback=_DEFAULT_VALUE): """ Create a new ``TLSConfiguration``, overriding some of the settings on the original configuration with the new settings. """ if validate_certificates is _DEFAULT_VALUE: validate_certificates = self.validate_certificates if certificate_chain is _DEFAULT_VALUE: certificate_chain = self.certificate_chain if ciphers is _DEFAULT_VALUE: ciphers = self.ciphers if inner_protocols is _DEFAULT_VALUE: inner_protocols = self.inner_protocols if lowest_supported_version is _DEFAULT_VALUE: lowest_supported_version = self.lowest_supported_version if highest_supported_version is _DEFAULT_VALUE: highest_supported_version = self.highest_supported_version if trust_store is _DEFAULT_VALUE: trust_store = self.trust_store if sni_callback is _DEFAULT_VALUE: sni_callback = self.sni_callback return self.__class__( validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) Context ~~~~~~~ The ``Context`` abstract base class defines an object that allows configuration of TLS. It can be thought of as a factory for ``TLSWrappedSocket`` and ``TLSWrappedBuffer`` objects. As much as possible implementers should aim to make these classes immutable: that is, they should prefer not to allow users to mutate their internal state directly, instead preferring to create new contexts from new TLSConfiguration objects. Obviously, the ABCs cannot enforce this constraint, and so they do not attempt to. The ``Context`` abstract base class has the following class definition:: TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] class _BaseContext(metaclass=ABCMeta): @abstractmethod def __init__(self, configuration: TLSConfiguration): """ Create a new context object from a given TLS configuration. """ @property @abstractmethod def configuration(self) -> TLSConfiguration: """ Returns the TLS configuration that was used to create the context. """ class ClientContext(_BaseContext): @abstractmethod def wrap_socket(self, socket: socket.socket, auto_handshake=True: bool, server_hostname=None: Optional[str]) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.connect()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. The optional parameter ``server_hostname`` specifies the hostname of the service which we are connecting to. This allows a single server to host multiple SSL-based services with distinct certificates, quite similarly to HTTP virtual hosts. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any, server_hostname=None: Optional[str]) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. The ``server_hostname`` parameter has the same meaning as in ``wrap_socket``. """ class ServerContext(_BaseContext): @abstractmethod def wrap_socket(self, socket: socket.socket, auto_handshake=True: bool) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.connect()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. """ Socket ~~~~~~ The socket-wrapper ABC will be defined by the ``TLSWrappedSocket`` ABC, which has the following definition:: class TLSWrappedSocket(metaclass=ABCMeta): # The various socket methods all must be implemented. Their definitions # have been elided from this class defintion in the PEP because they # aren't instructive. @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[CipherSuite]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ @abstractmethod def unwrap(self) -> socket.socket: """ Cleanly terminate the TLS connection on this wrapped socket. Once called, this ``TLSWrappedSocket`` can no longer be used to transmit data. Returns the socket that was wrapped with TLS. """ Buffer ~~~~~~ The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, which has the following definition:: class TLSWrappedBuffer(metaclass=ABCMeta): @abstractmethod def read(self, amt=None: int) -> bytes: """ Read up to ``amt`` bytes of data from the input buffer and return the result as a ``bytes`` instance. If ``amt`` is ``None``, will attempt to read until either EOF is reached or until further attempts to read would raise either ``WantReadError`` or ``WantWriteError``. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``read()`` can also cause write operations. """ @abstractmethod def read_into(self, buffer: Any, amt=None: int) -> int: """ Read up to ``amt`` bytes of data from the input buffer into ``buffer``, which must be an object that implements the buffer protocol. Returns the number of bytes read. If ``amt`` is ``None``, will attempt to read until either EOF is reached or until further attempts to read would raise either ``WantReadError`` or ``WantWriteError``, or until the buffer is full. Raises ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``read_into()`` can also cause write operations. """ @abstractmethod def write(self, buf: Any) -> int: """ Write ``buf`` in encrypted form to the output buffer and return the number of bytes written. The ``buf`` argument must be an object supporting the buffer interface. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``write()`` can also cause read operations. """ @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[CipherSuite]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ @abstractmethod def shutdown(self) -> None: """ Performs a clean TLS shut down. This should generally be used whenever possible to signal to the remote peer that the content is finished. """ Cipher Suites ~~~~~~~~~~~~~ Supporting cipher suites in a truly library-agnostic fashion is a remarkably difficult undertaking. Different TLS implementations often have *radically* different APIs for specifying cipher suites, but more problematically these APIs frequently differ in capability as well as in style. Some examples are shown below: OpenSSL ^^^^^^^ OpenSSL uses a well-known cipher string format. This format has been adopted as a configuration language by most products that use OpenSSL, including Python. This format is relatively easy to read, but has a number of downsides: it is a string, which makes it remarkably easy to provide bad inputs; it lacks much detailed validation, meaning that it is possible to configure OpenSSL in a way that doesn't allow it to negotiate any cipher at all; and it allows specifying cipher suites in a number of different ways that make it tricky to parse. The biggest problem with this format is that there is no formal specification for it, meaning that the only way to parse a given string the way OpenSSL would is to get OpenSSL to parse it. OpenSSL's cipher strings can look like this: 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5' This string demonstrates some of the complexity of the OpenSSL format. For example, it is possible for one entry to specify multiple cipher suites: the entry ``ECDH+AESGCM`` means "all ciphers suites that include both elliptic-curve Diffie-Hellman key exchange and AES in Galois Counter Mode". More explicitly, that will expand to four cipher suites: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" That makes parsing a complete OpenSSL cipher string extremely tricky. Add to the fact that there are other meta-characters, such as "!" (exclude all cipher suites that match this criterion, even if they would otherwise be included: "!MD5" means that no cipher suites using the MD5 hash algorithm should be included), "-" (exclude matching ciphers if they were already included, but allow them to be re-added later if they get included again), and "+" (include the matching ciphers, but place them at the end of the list), and you get an *extremely* complex format to parse. On top of this complexity it should be noted that the actual result depends on the OpenSSL version, as an OpenSSL cipher string is valid so long as it contains at least one cipher that OpenSSL recognises. OpenSSL also uses different names for its ciphers than the names used in the relevant specifications. See the manual page for ``ciphers(1)`` for more details. The actual API inside OpenSSL for the cipher string is simple: char *cipher_list = ; int rc = SSL_CTX_set_cipher_list(context, cipher_list); This means that any format that is used by this module must be able to be converted to an OpenSSL cipher string for use with OpenSSL. SecureTransport ^^^^^^^^^^^^^^^ SecureTransport is the macOS system TLS library. This library is substantially more restricted than OpenSSL in many ways, as it has a much more restricted class of users. One of these substantial restrictions is in controlling supported cipher suites. Ciphers in SecureTransport are represented by a C ``enum``. This enum has one entry per cipher suite, with no aggregate entries, meaning that it is not possible to reproduce the meaning of an OpenSSL cipher string like "ECDH+AESGCM" without hand-coding which categories each enum member falls into. However, the names of most of the enum members are in line with the formal names of the cipher suites: that is, the cipher suite that OpenSSL calls "ECDHE-ECDSA-AES256-GCM-SHA384" is called "TLS_ECDHE_ECDHSA_WITH_AES_256_GCM_SHA384" in SecureTransport. The API for configuring cipher suites inside SecureTransport is simple: SSLCipherSuite ciphers[] = {TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ...}; OSStatus status = SSLSetEnabledCiphers(context, ciphers, sizeof(cphers)); SChannel ^^^^^^^^ SChannel is the Windows system TLS library. SChannel has extremely restrictive support for controlling available TLS cipher suites, and additionally adopts a third method of expressing what TLS cipher suites are supported. Specifically, SChannel defines a set of ``ALG_ID`` constants (C unsigned ints). Each of these constants does not refer to an entire cipher suite, but instead an individual algorithm. Some examples are ``CALG_3DES`` and ``CALG_AES_256``, which refer to the bulk encryption algorithm used in a cipher suite, ``CALG_DH_EPHEM`` and ``CALG_RSA_KEYX`` which refer to part of the key exchange algorithm used in a cipher suite, ``CALG_SHA1`` and ``CALG_MD5`` which refer to the message authentication code used in a cipher suite, and ``CALG_ECDSA`` and ``CALG_RSA_SIGN`` which refer to the signing portions of the key exchange algorithm. This can be thought of as the half of OpenSSL's functionality that SecureTransport doesn't have: SecureTransport only allows specifying exact cipher suites, while SChannel only allows specifying *parts* of the cipher suite, while OpenSSL allows both. Determining which cipher suites are allowed on a given connection is done by providing a pointer to an array of these ``ALG_ID`` constants. This means that any suitable API must allow the Python code to determine which ``ALG_ID`` constants must be provided. Proposed Interface ^^^^^^^^^^^^^^^^^^ The proposed interface for the new module is influenced by the combined set of limitations of the above implementations. Specifically, as every implementation *except* OpenSSL requires that each individual cipher be provided, there is no option but to provide that lowest-common denominator approach. The simplest approach is to provide an enumerated type that includes all of the cipher suites defined for TLS. The values of the enum members will be their two-octet cipher identifier as used in the TLS handshake, stored as a tuple of integers. The names of the enum members will be their IANA-registered cipher suite names. Rather than populate this enum by hand, it is likely that we'll define a script that can build it from Christian Heimes' `tlsdb JSON file`_ (warning: large file). This also opens up the possibility of extending the API with additional querying function, such as determining which TLS versions support which ciphers, if that functionality is found to be useful or necessary. If users find this approach to be onerous, a future extension to this API can provide helpers that can reintroduce OpenSSL's aggregation functionality. Because this enum would be enormous, the entire enum is not provided here. Instead, a small sample of entries is provided to give a flavor of how it will appear. class CipherSuite(Enum): ... TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA = (0xC0, 0x12) ... TLS_ECDHE_ECDSA_WITH_AES_128_CCM = (0xC0, 0xAC) ... TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 = (0xC0, 0x2B) ... Protocol Negotiation ~~~~~~~~~~~~~~~~~~~~ Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 handshake. While NPN and ALPN are, at their fundamental level, built on top of bytestrings, string-based APIs are frequently problematic as they allow for errors in typing that can be hard to detect. For this reason, this module would define a type that protocol negotiation implementations can pass and be passed. This type would wrap a bytestring to allow for aliases for well-known protocols. This allows us to avoid the problems inherent in typos for well-known protocols, while allowing the full extensibility of the protocol negotiation layer if needed by letting users pass byte strings directly. :: class NextProtocol(Enum): H2 = b'h2' H2C = b'h2c' HTTP1 = b'http/1.1' WEBRTC = b'webrtc' C_WEBRTC = b'c-webrtc' FTP = b'ftp' STUN = b'stun.nat-discovery' TURN = b'stun.turn' TLS Versions ~~~~~~~~~~~~ It is often useful to be able to restrict the versions of TLS you're willing to support. There are many security advantages in refusing to use old versions of TLS, and some misbehaving servers will mishandle TLS clients advertising support for newer versions. The following enumerated type can be used to gate TLS versions. Forward-looking applications should almost never set a maximum TLS version unless they absolutely must, as a TLS backend that is newer than the Python that uses it may support TLS versions that are not in this enumerated type. Additionally, this enumerated type defines two additional flags that can always be used to request either the lowest or highest TLS version supported by an implementation. :: class TLSVersion(Enum): MINIMUM_SUPPORTED SSLv2 SSLv3 TLSv1 TLSv1_1 TLSv1_2 TLSv1_3 MAXIMUM_SUPPORTED Errors ~~~~~~ This module would define three base classes for use with error handling. Unlike the other classes defined here, these classes are not *abstract*, as they have no behaviour. They exist simply to signal certain common behaviours. Backends should subclass these exceptions in their own packages, but needn't define any behaviour for them. In general, concrete implementations should subclass these exceptions rather than throw them directly. This makes it moderately easier to determine which concrete TLS implementation is in use during debugging of unexpected errors. However, this is not mandatory. The definitions of the errors are below:: class TLSError(Exception): """ The base exception for all TLS related errors from any backend. Catching this error should be sufficient to catch *all* TLS errors, regardless of what backend is used. """ class WantWriteError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is written to the network, or until the output buffer is drained. """ class WantReadError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is read from the network, or until more data is available in the input buffer. """ Certificates ~~~~~~~~~~~~ This module would define an abstract X509 certificate class. This class would have almost no behaviour, as the goal of this module is not to provide all possible relevant cryptographic functionality that could be provided by X509 certificates. Instead, all we need is the ability to signal the source of a certificate to a concrete implementation. For that reason, this certificate implementation defines only constructors. In essence, the certificate object in this module could be as abstract as a handle that can be used to locate a specific certificate. Concrete implementations may choose to provide alternative constructors, e.g. to load certificates from HSMs. If a common interface emerges for doing this, this module may be updated to provide a standard constructor for this use-case as well. Concrete implementations should aim to have Certificate objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. class Certificate(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes) -> Certificate: """ Creates a Certificate object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN CERTIFICATE" and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, AnyStr]) -> Certificate: """ Creates a Certificate object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. """ Private Keys ~~~~~~~~~~~~ This module would define an abstract private key class. Much like the Certificate class, this class has almost no behaviour in order to give as much freedom as possible to the concrete implementations to treat keys carefully. This class has all the caveats of the ``Certificate`` class. class PrivateKey(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes, password=None: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: """ Creates a PrivateKey object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN", the key type, and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. The key may additionally be encrypted. If it is, the ``password`` argument can be used to decrypt the key. The ``password`` argument may be a function to call to get the password for decrypting the private key. It will only be called if the private key is encrypted and a password is necessary. It will be called with no arguments, and it should return either bytes or bytearray containing the password. Alternatively a bytes, or bytearray value may be supplied directly as the password argument. It will be ignored if the private key is not encrypted and no password is needed. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, bytes, str], password=None: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: """ Creates a PrivateKey object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. The ``password`` parameter behaves exactly as the equivalent parameter on ``from_buffer``. """ Trust Store ~~~~~~~~~~~ As discussed above, loading a trust store represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. For this reason, we need to provide a model that assumes very little about the form that trust stores take. This problem is the same as the one that the Certificate and PrivateKey types need to solve. For this reason, we use the exact same model, by creating an opaque type that can encapsulate the various means that TLS backends may open a trust store. A given TLS implementation is not required to implement all of the constructors. However, it is strongly recommended that a given TLS implementation provide the ``system`` constructor if at all possible, as this is the most common validation trust store that is used. Concrete implementations may also add their own constructors. Concrete implementations should aim to have TrustStore objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. class TrustStore(metaclass=ABCMeta): @abstractclassmethod def system(cls) -> TrustStore: """ Returns a TrustStore object that represents the system trust database. """ @abstractclassmethod def from_pem_file(cls, path: Union[pathlib.Path, bytes, str]) -> TrustStore: """ Initializes a trust store from a single file full of PEMs. """ Changes to the Standard Library =============================== The portions of the standard library that interact with TLS should be revised to use these ABCs. This will allow them to function with other TLS backends. This includes the following modules: - asyncio - ftplib - http.client - imaplib - nntplib - poplib - smtplib Future ====== Major future TLS features may require revisions of these ABCs. These revisions should be made cautiously: many backends may not be able to move forward swiftly, and will be invalidated by changes in these ABCs. This is acceptable, but wherever possible features that are specific to individual implementations should not be added to the ABCs. The ABCs should restrict themselves to high-level descriptions of IETF-specified features. ToDo ==== * Consider adding a new parameter (``valid_subjects``?) to ``wrap_socket`` and ``wrap_buffers`` that specifies in a *typed* manner what kind of entries in the SAN field are acceptable. This would break the union between SNI and cert validation, which may be a good thing (you can't SNI an IP address, but you can validate a cert with one if you want). * It's annoying that there's no type signature for fileobj. Do I really have to define one as part of this PEP? Otherwise, how do I define the types of the arguments to ``wrap_buffers``? * Do we need ways to control hostname validation? * Do we need to support getpeercert? Should we always return DER instead of the weird semi-structured thing? * How do we load certs from locations on disk? What about HSMs? * How do we signal to load certs from the OS? What happens if an implementation doesn't let you *not* load those certs? References ========== .. _ssl module: https://docs.python.org/3/library/ssl.html .. _OpenSSL Library: https://www.openssl.org/ .. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ .. _certifi: https://pypi.org/project/certifi/ .. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext .. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket .. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject .. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError .. _MSDN articles: https://msdn.microsoft.com/en-us/library/windows/desktop/mt490158(v=vs.85).aspx .. _tlsdb JSON file: https://github.com/tiran/tlsdb/blob/master/tlsdb.json Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From wes.turner at gmail.com Fri Jan 20 00:41:49 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 19 Jan 2017 23:41:49 -0600 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: On Thursday, January 19, 2017, Cory Benfield wrote: > > Configuration > ~~~~~~~~~~~~~ > > The ``TLSConfiguration`` concrete class defines an object that can hold and > manage TLS configuration. The goals of this class are as follows: > > 1. To provide a method of specifying TLS configuration that avoids the > risk of > errors in typing (this excludes the use of a simple dictionary). > 2. To provide an object that can be safely compared to other configuration > objects to detect changes in TLS configuration, for use with the SNI > callback. > > This class is not an ABC, primarily because it is not expected to have > implementation-specific behaviour. The responsibility for transforming a > ``TLSConfiguration`` object into a useful set of configuration for a given > TLS > implementation belongs to the Context objects discussed below. > > This class has one other notable property: it is immutable. This is a > desirable > trait for a few reasons. The most important one is that it allows these > objects > to be used as dictionary keys, which is potentially extremely valuable for > certain TLS backends and their SNI configuration. On top of this, it frees > implementations from needing to worry about their configuration objects > being > changed under their feet, which allows them to avoid needing to carefully > synchronize changes between their concrete data structures and the > configuration object. > > The ``TLSConfiguration`` object would be defined by the following code: > > ServerNameCallback = Callable[[TLSBufferObject, Optional[str], > TLSConfiguration], Any] > > > _configuration_fields = [ > 'validate_certificates', > 'certificate_chain', > 'ciphers', > 'inner_protocols', > 'lowest_supported_version', > 'highest_supported_version', > 'trust_store', > 'sni_callback', > ] Thanks! TLSConfiguration looks much easier to review; and should make other implementations easier. I read a great (illustrated) intro to TLS1.3 the other day: https://temen.io/resources/tls-gets-an-upgrade:-welcome-1.3/ - 1-RTT and 0-RTT look useful. - There's a reduced set of cipher suites https://tlswg.github.io/tls13-spec/ #rfc.section.4.3 - Are there additional parameters relevant to TLS1.3 for the TLSConfiguration object? - If necessary, how should TLSConfiguration parameter fields be added? > > > _DEFAULT_VALUE = object() > > > class TLSConfiguration(namedtuple('TLSConfiguration', > _configuration_fields)): > """ > An imutable TLS Configuration object. This object has the following > properties: > > :param validate_certificates bool: Whether to validate the TLS > certificates. This switch operates at a very broad scope: > either > validation is enabled, in which case all forms of validation > are > performed including hostname validation if possible, or > validation > is disabled, in which case no validation is performed. > > Not all backends support having their certificate validation > disabled. If a backend does not support having their > certificate > validation disabled, attempting to set this property to > ``False`` > will throw a ``TLSError`` when this object is passed into a > context object. > > :param certificate_chain Tuple[Tuple[Certificate],PrivateKey]: The > certificate, intermediate certificate, and the corresponding > private key for the leaf certificate. These certificates will > be > offered to the remote peer during the handshake if required. > > The first Certificate in the list must be the leaf > certificate. All > subsequent certificates will be offered as intermediate > additional > certificates. > > :param ciphers Tuple[CipherSuite]: > The available ciphers for TLS connections created with this > configuration, in priority order. > > :param inner_protocols Tuple[Union[NextProtocol, bytes]]: > Protocols that connections created with this configuration > should > advertise as supported during the TLS handshake. These may be > advertised using either or both of ALPN or NPN. This list of > protocols should be ordered by preference. > > :param lowest_supported_version TLSVersion: > The minimum version of TLS that should be allowed on TLS > connections using this configuration. > > :param highest_supported_version TLSVersion: > The maximum version of TLS that should be allowed on TLS > connections using this configuration. > > :param trust_store TrustStore: > The trust store that connections using this configuration will > use > to validate certificates. > > :param sni_callback Optional[ServerNameCallback]: > A callback function that will be called after the TLS Client > Hello > handshake message has been received by the TLS server when the > TLS > client specifies a server name indication. > > Only one callback can be set per ``TLSConfiguration``. If the > ``sni_callback`` is ``None`` then the callback is disabled. If > the > ``TLSConfiguration`` is used for a ``ClientContext`` then this > setting will be ignored. > > The ``callback`` function will be called with three arguments: > the > first will be the ``TLSBufferObject`` for the connection; the > second will be a string that represents the server name that > the > client is intending to communicate (or ``None`` if the TLS > Client > Hello does not contain a server name); and the third argument > will > be the original ``Context``. The server name argument will be > the > IDNA *decoded* server name. > > The ``callback`` must return a ``TLSConfiguration`` to allow > negotiation to continue. Other return values signal errors. > Attempting to control what error is signaled by the underlying > TLS > implementation is not specified in this API, but is up to the > concrete implementation to handle. > > The Context will do its best to apply the ``TLSConfiguration`` > changes from its original configuration to the incoming > connection. > This will usually include changing the certificate chain, but > may > also include changes to allowable ciphers or any other > configuration settings. > """ > __slots__ = () > > def __new__(cls, validate_certificates=None: Optional[bool], > certificate_chain=None: Optional[Tuple[Tuple[Certificate], > PrivateKey]], > ciphers=None: Optional[Tuple[CipherSuite]], > inner_protocols=None: Optional[Tuple[Union[NextProtocol, > bytes]]], > lowest_supported_version=None: > Optional[TLSVersion], > highest_supported_version=None: > Optional[TLSVersion], > trust_store=None: Optional[TrustStore], > sni_callback=None: Optional[ServerNameCallback]): > > if validate_certificates is None: > validate_certificates = True > > if ciphers is None: > ciphers = DEFAULT_CIPHER_LIST > > if inner_protocols is None: > inner_protocols = [] > > if lowest_supported_version is None: > lowest_supported_version = TLSVersion.TLSv1 > > if highest_supported_version is None: > highest_supported_version = TLSVersion.MAXIMUM_SUPPORTED > > return super().__new__( > cls, validate_certificates, certificate_chain, ciphers, > inner_protocols, lowest_supported_version, > highest_supported_version, trust_store, sni_callback > ) > > def update(self, validate_certificates=_DEFAULT_VALUE, > certificate_chain=_DEFAULT_VALUE, > ciphers=_DEFAULT_VALUE, > inner_protocols=_DEFAULT_VALUE, > lowest_supported_version=_DEFAULT_VALUE, > highest_supported_version=_DEFAULT_VALUE, > trust_store=_DEFAULT_VALUE, > sni_callback=_DEFAULT_VALUE): > """ > Create a new ``TLSConfiguration``, overriding some of the > settings > on the original configuration with the new settings. > """ > if validate_certificates is _DEFAULT_VALUE: > validate_certificates = self.validate_certificates > > if certificate_chain is _DEFAULT_VALUE: > certificate_chain = self.certificate_chain > > if ciphers is _DEFAULT_VALUE: > ciphers = self.ciphers > > if inner_protocols is _DEFAULT_VALUE: > inner_protocols = self.inner_protocols > > if lowest_supported_version is _DEFAULT_VALUE: > lowest_supported_version = self.lowest_supported_version > > if highest_supported_version is _DEFAULT_VALUE: > highest_supported_version = self.highest_supported_version > > if trust_store is _DEFAULT_VALUE: > trust_store = self.trust_store > > if sni_callback is _DEFAULT_VALUE: > sni_callback = self.sni_callback > > return self.__class__( > validate_certificates, certificate_chain, ciphers, > inner_protocols, lowest_supported_version, > highest_supported_version, trust_store, sni_callback > ) > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Fri Jan 20 08:46:04 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 20 Jan 2017 13:46:04 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: > On 20 Jan 2017, at 04:38, Wes Turner wrote: > > Thanks! TLSConfiguration looks much easier to review; and should make other implementations easier. > > I read a great (illustrated) intro to TLS1.3 the other day: > > https://temen.io/resources/tls-gets-an-upgrade:-welcome-1.3/ > > - 1-RTT and 0-RTT look useful. > - There's a reduced set of cipher suites > > https://tlswg.github.io/tls13-spec/ #rfc.section.4.3 > > - Are there additional parameters relevant to TLS1.3 for the TLSConfiguration object? > - If necessary, how should TLSConfiguration parameter fields be added? So both 0-RTT and 1-RTT have little to no library support at this time. Certainly the three main implementations that this proposal considers (OpenSSL, SChannel, SecureTransport) have no public APIs to control this kind of functionality. Indeed, none of those implementations is shipping TLSv1.3 yet as far as I?m aware. OpenSSL has a date by which they plan to have TLSv1.3 support complete, and based on Apple?s adoption of TLS tech I?d expect to see TLSv1.3 in SecureTransport within 18 months and possibly as soon as September. The TL;DR here is that I think we should not attempt to spec APIs for 1-RTT and 0-RTT yet, until we understand how implementations plan to support it. As to the reduced set of cipher suites, all either do or will have IANA assigned names, and so can be managed as per the cipher enum. As to adding TLSConfiguration parameter fields, I think they are subject to the regular standard library update process. Generally speaking we should have a policy that adding fields should be a difficult thing to do unless there is substantial need for their addition, to avoid making the object almost impossible to manage. However, otherwise the regular standard library concerns should all apply. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Jan 20 11:30:09 2017 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 20 Jan 2017 10:30:09 -0600 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: On Friday, January 20, 2017, Cory Benfield wrote: > > On 20 Jan 2017, at 04:38, Wes Turner > wrote: > > Thanks! TLSConfiguration looks much easier to review; and should make > other implementations easier. > > I read a great (illustrated) intro to TLS1.3 the other day: > > https://temen.io/resources/tls-gets-an-upgrade:-welcome-1.3/ > > - 1-RTT and 0-RTT look useful. > - There's a reduced set of cipher suites > > https://tlswg.github.io/tls13-spec/ #rfc.section.4.3 > > - Are there additional parameters relevant to TLS1.3 for the > TLSConfiguration object? > - If necessary, how should TLSConfiguration parameter fields be added? > >> > So both 0-RTT and 1-RTT have little to no library support at this time. > Certainly the three main implementations that this proposal considers > (OpenSSL, SChannel, SecureTransport) have no public APIs to control this > kind of functionality. Indeed, none of those implementations is shipping > TLSv1.3 yet as far as I?m aware. OpenSSL has a date by which they plan to > have TLSv1.3 support complete, and based on Apple?s adoption of TLS tech > I?d expect to see TLSv1.3 in SecureTransport within 18 months and possibly > as soon as September. The TL;DR here is that I think we should not attempt > to spec APIs for 1-RTT and 0-RTT yet, until we understand how > implementations plan to support it. > I think there's somewhat high likelihood that additional parameters will be relevant to TLS Configuration in the future. > > As to the reduced set of cipher suites, all either do or will have IANA > assigned names, and so can be managed as per the cipher enum. > > As to adding TLSConfiguration parameter fields, I think they are subject > to the regular standard library update process. Generally speaking we > should have a policy that adding fields should be a difficult thing to do > unless there is substantial need for their addition, to avoid making the > object almost impossible to manage. > So the interface contract of the TLS configuration namedtuple is that there are, minimally: - .attributes for each of tls._configuration_fields - .update() - __new__() # instantiation-time validation IIUC, in order to validate a given configuration, the option is to: - subclass TLSConfiguration, define __new__(), and call super() So the normal standard library practice of duck-typing (and not explicitly checking for TLSConfiguration in cls.__bases__) does or does not exclude the use of an alternate configuration object which satisfies the interface contract? Is immutability a hard requirement of alternate/future implementations? > However, otherwise the regular standard library concerns should all apply. > > Cory > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Fri Jan 20 11:42:08 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 20 Jan 2017 16:42:08 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: > On 20 Jan 2017, at 16:30, Wes Turner wrote: > > I think there's somewhat high likelihood that additional parameters will be relevant to TLS Configuration in the future. I think that we can upgrade ?somewhat high likelihood? to ?mathematical certainty?. This again fits with my overall position, which is not to pitch this as the completed solution but instead as a good starting point for conservative evolution. > As to the reduced set of cipher suites, all either do or will have IANA assigned names, and so can be managed as per the cipher enum. > > As to adding TLSConfiguration parameter fields, I think they are subject to the regular standard library update process. Generally speaking we should have a policy that adding fields should be a difficult thing to do unless there is substantial need for their addition, to avoid making the object almost impossible to manage. > > So the interface contract of the TLS configuration namedtuple is that there are, minimally: > > - .attributes for each of tls._configuration_fields > - .update() > - __new__() # instantiation-time validation Correct. Note that this isn?t an ABC like many of the other classes, but a concrete implementation: it is not generally expected that TLS implementations will want to add behaviours to this object, or to extend it directly. > IIUC, in order to validate a given configuration, the option is to: > > - subclass TLSConfiguration, define __new__(), and call super() That shouldn?t be needed: just write a function that accepts one and look at the values. > So the normal standard library practice of duck-typing (and not explicitly checking for TLSConfiguration in cls.__bases__) does or does not exclude the use of an alternate configuration object which satisfies the interface contract? Does not exclude a duck-typed object. > Is immutability a hard requirement of alternate/future implementations? It?s hard to make immutability a hard requirement because there is no enforcement mechanism for it. However, I think the appropriate caveat is that if some future or alternate config object is mutable, and someone mutates it after passing it to a Context, the outcome of that change is undefined by this abstract API. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Jan 20 13:22:12 2017 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 20 Jan 2017 12:22:12 -0600 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: Got it. Thanks again! On Friday, January 20, 2017, Cory Benfield wrote: > > On 20 Jan 2017, at 16:30, Wes Turner > wrote: > > I think there's somewhat high likelihood that additional parameters will > be relevant to TLS Configuration in the future. > > > I think that we can upgrade ?somewhat high likelihood? to ?mathematical > certainty?. This again fits with my overall position, which is not to pitch > this as the completed solution but instead as a good starting point for > conservative evolution. > > As to the reduced set of cipher suites, all either do or will have IANA >> assigned names, and so can be managed as per the cipher enum. >> >> As to adding TLSConfiguration parameter fields, I think they are subject >> to the regular standard library update process. Generally speaking we >> should have a policy that adding fields should be a difficult thing to do >> unless there is substantial need for their addition, to avoid making the >> object almost impossible to manage. >> > > So the interface contract of the TLS configuration namedtuple is that > there are, minimally: > > - .attributes for each of tls._configuration_fields > - .update() > - __new__() # instantiation-time validation > > > Correct. Note that this isn?t an ABC like many of the other classes, but a > concrete implementation: it is not generally expected that TLS > implementations will want to add behaviours to this object, or to extend it > directly. > > IIUC, in order to validate a given configuration, the option is to: > > - subclass TLSConfiguration, define __new__(), and call super() > > > That shouldn?t be needed: just write a function that accepts one and look > at the values. > > So the normal standard library practice of duck-typing (and not explicitly > checking for TLSConfiguration in cls.__bases__) does or does not exclude > the use of an alternate configuration object which satisfies the interface > contract? > > > Does not exclude a duck-typed object. > > Is immutability a hard requirement of alternate/future implementations? > > > It?s hard to make immutability a hard requirement because there is no > enforcement mechanism for it. However, I think the appropriate caveat is > that if some future or alternate config object is mutable, and someone > mutates it after passing it to a Context, the outcome of that change is > undefined by this abstract API. > > Cory > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Jan 20 18:00:58 2017 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 20 Jan 2017 15:00:58 -0800 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: [resending from a real computer since the security-sig auto(?)-moderator didn't like me top-posting from my phone :-)] On Thu, Jan 19, 2017 at 9:29 AM, Cory Benfield wrote: > All, > > Thanks for your feedback on the draft PEP I proposed last week! There was a lot of really enthusiastic and valuable feedback both on this mailing list and on GitHub. > > I believe I?ve addressed a lot of the concerns that were brought up with the PEP now, so I?d like to ask that interested parties take another look. Hi Cory, Great work! A few quick thoughts: - given that for next protocol negotiation we have to accept arbitrary bytestrings and the primary value of the NextProtocol class is to protect against typos, I wonder if it would simplify things slightly to make the attributes of this class just *be* bytestrings. I.e. what you have now but without the inheritance from enum. - what object types *do* you expect to be passed to wrap_buffers? I was assuming bytearrays, but the text at the bottom suggests file-likes? - a minor thing that I think is very important for usability is that there be some way to specify a name which library we're using with a single object. For example, curio has a high-level helper that abstracts over a lot of boilerplate in setting up a listening socket: await curio.tcp_server(host, port, client_connected_task, *, family=AF_INET, backlog=100, ssl=None, reuse_address=True) (Here ssl= is an SSLContext. Actually it's a curio.ssl.SSLContext -- see https://github.com/dabeaz/curio/blob/master/curio/ssl.py) Asyncio is similar: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.AbstractEventLoop.create_connection It isn't quite clear to me right now how this kind of API should look with your proposal. Obviously they can't just take a SSLConfiguration object, because there's no way to look at an SSLConfiguration object and figure out what backend is in use (even though in general a given SSLConfiguration is only usable with a specific backend, because backends provide the Cert type etc). They could take a ServerContext/ClientContext, I guess? But it would be nice if there were some way to say "give me the default configuration, using SChannel". Or to write a function that sets up an SSLConfiguration while being generic over backends, so like it takes the backend as an argument and then uses that backend's cert type etc. Maybe the solution is to require that each implementation provide a namespace where each of its concrete types are given standard names? So like I know schannel.ClientContext is the schannel implementation of the tlsabc.ClientContext interface? -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Sat Jan 21 08:07:25 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Jan 2017 00:07:25 +1100 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: On 20 January 2017 at 04:29, Cory Benfield wrote: > Please let me know what you think. > > The version of the draft PEP, from commit ce74bc60, is reproduced below. Thanks Cory, this is looking really good. I don't have anything to add on the security design front, but do have a few comments/questions on the API design and explanation front. > Configuration > ~~~~~~~~~~~~~ > > The ``TLSConfiguration`` concrete class defines an object that can hold and > manage TLS configuration. The goals of this class are as follows: > > 1. To provide a method of specifying TLS configuration that avoids the risk of > errors in typing (this excludes the use of a simple dictionary). > 2. To provide an object that can be safely compared to other configuration > objects to detect changes in TLS configuration, for use with the SNI > callback. > > This class is not an ABC, primarily because it is not expected to have > implementation-specific behaviour. The responsibility for transforming a > ``TLSConfiguration`` object into a useful set of configuration for a given TLS > implementation belongs to the Context objects discussed below. > > This class has one other notable property: it is immutable. This is a desirable > trait for a few reasons. The most important one is that it allows these objects > to be used as dictionary keys, which is potentially extremely valuable for > certain TLS backends and their SNI configuration. On top of this, it frees > implementations from needing to worry about their configuration objects being > changed under their feet, which allows them to avoid needing to carefully > synchronize changes between their concrete data structures and the > configuration object. > > The ``TLSConfiguration`` object would be defined by the following code: > > ServerNameCallback = Callable[[TLSBufferObject, Optional[str], TLSConfiguration], Any] > > > _configuration_fields = [ > 'validate_certificates', > 'certificate_chain', > 'ciphers', > 'inner_protocols', > 'lowest_supported_version', > 'highest_supported_version', > 'trust_store', > 'sni_callback', > ] > > > _DEFAULT_VALUE = object() > > > class TLSConfiguration(namedtuple('TLSConfiguration', _configuration_fields)): I agree with Wes that the backwards compatibility guarantees around adding new configuration fields should be clarified. I think it will suffice to say that "new fields are only appended, existing fields are never removed, renamed, or reordered". That means that: - tuple unpacking will be forward compatible as long as you use *args at the end - numeric lookup will be forward compatible That doesn't make either of them a good idea (vs just using attribute lookups), but it does provide an indication to future maintainers that such code shouldn't be gratuitously broken either. > Context > ~~~~~~~ > > The ``Context`` abstract base class defines an object that allows configuration > of TLS. It can be thought of as a factory for ``TLSWrappedSocket`` and > ``TLSWrappedBuffer`` objects. > > As much as possible implementers should aim to make these classes immutable: > that is, they should prefer not to allow users to mutate their internal state > directly, instead preferring to create new contexts from new TLSConfiguration > objects. Obviously, the ABCs cannot enforce this constraint, and so they do not > attempt to. > > The ``Context`` abstract base class has the following class definition:: This intro section talks about a combined "Context" objection, but the implementation has been split into ServerContext and ClientContext. That split could also use some explanation in the background section of the PEP. > Proposed Interface > ^^^^^^^^^^^^^^^^^^ > > The proposed interface for the new module is influenced by the combined set of > limitations of the above implementations. Specifically, as every implementation > *except* OpenSSL requires that each individual cipher be provided, there is no > option but to provide that lowest-common denominator approach. The second sentence here doesn't match the description of SChannel cipher configuration, so I'm not clear on how the proposed interface would map to an SChannel backend. > Errors > ~~~~~~ > > This module would define three base classes for use with error handling. Unlike > the other classes defined here, these classes are not *abstract*, as they have > no behaviour. They exist simply to signal certain common behaviours. Backends > should subclass these exceptions in their own packages, but needn't define any > behaviour for them. > > In general, concrete implementations should subclass these exceptions rather > than throw them directly. This makes it moderately easier to determine which > concrete TLS implementation is in use during debugging of unexpected errors. > However, this is not mandatory. This is the one part of the PEP that I think may need to discuss transition strategies for libraries and frameworks that currently let ssl module exceptions escape to their users: how do they do that in a way that's transparent to API consumers that currently capture the ssl module exceptions? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Sun Jan 22 07:01:00 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Sun, 22 Jan 2017 12:01:00 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: <36AFC91B-7D47-4DA4-966A-926185B5B0B2@lukasa.co.uk> > On 20 Jan 2017, at 23:00, Nathaniel Smith wrote: > > - given that for next protocol negotiation we have to accept arbitrary > bytestrings and the primary value of the NextProtocol class is to > protect against typos, I wonder if it would simplify things slightly > to make the attributes of this class just *be* bytestrings. I.e. what > you have now but without the inheritance from enum. While we *could*, I don?t think the value-add of doing this is very high. Basically the only thing it simplifies is the type declaration, and I don?t know that it?s worth doing that. ;) > - what object types *do* you expect to be passed to wrap_buffers? I > was assuming bytearrays, but the text at the bottom suggests > file-likes? Yeah, good question. I was assuming file-likes as well, but I don?t see any reason we couldn?t also do bytearrays. What do you think the advantage of that is? > It isn't quite clear to me right now how this kind of API should look > with your proposal. Obviously they can't just take a SSLConfiguration > object, because there's no way to look at an SSLConfiguration object > and figure out what backend is in use (even though in general a given > SSLConfiguration is only usable with a specific backend, because > backends provide the Cert type etc). They could take a > ServerContext/ClientContext, I guess? But it would be nice if there > were some way to say "give me the default configuration, using > SChannel". Or to write a function that sets up an SSLConfiguration > while being generic over backends, so like it takes the backend as an > argument and then uses that backend's cert type etc. Yeah, I was assuming that they?d take a ClientContext/ServerContext object, rather than a configuration plus an instruction on which backend to use. It?s not clear to me that a function that setups up a configuration while generic over backends is actually a meaningful thing to have: are there really going to be users who are insistent on a specific TLS configuration but don?t care what concrete implementation is going to be used, such that their libraries have to pick it for them? Cory From cory at lukasa.co.uk Sun Jan 22 07:18:06 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Sun, 22 Jan 2017 12:18:06 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> Message-ID: <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> > On 21 Jan 2017, at 13:07, Nick Coghlan wrote: > > > I agree with Wes that the backwards compatibility guarantees around > adding new configuration fields should be clarified. > > I think it will suffice to say that "new fields are only appended, > existing fields are never removed, renamed, or reordered". That means > that: > > - tuple unpacking will be forward compatible as long as you use *args at the end > - numeric lookup will be forward compatible Good idea. > This intro section talks about a combined "Context" objection, but the > implementation has been split into ServerContext and ClientContext. > > That split could also use some explanation in the background section of the PEP. Good idea, I?ll expand on that. >> Proposed Interface >> ^^^^^^^^^^^^^^^^^^ >> >> The proposed interface for the new module is influenced by the combined set of >> limitations of the above implementations. Specifically, as every implementation >> *except* OpenSSL requires that each individual cipher be provided, there is no >> option but to provide that lowest-common denominator approach. > > The second sentence here doesn't match the description of SChannel > cipher configuration, so I'm not clear on how the proposed interface > would map to an SChannel backend. Yeah, this is a point I?m struggling with internally. SChannel?s API is frustratingly limited. The way I see it there are two options: 1. Allow the possibility that SChannel may allow ciphers that others do not given the same cipher configuration. This is not a good solution, frankly, because the situation in which this will happen is predominantly that SChannel will allow modes or key/hash sizes that the other implementations would forbid, given the same cipher configuration. 2. Force all other implementations to be as bad as SChannel: that is, to make it impossible to restrict key sizes and cipher modes on all implementations because SChannel can?t. I don?t really like either of those choices. I *think* 2 is worse than 1, but I?m not sure about that. People with opinions should really weigh in on it. > This is the one part of the PEP that I think may need to discuss > transition strategies for libraries and frameworks that currently let > ssl module exceptions escape to their users: how do they do that in a > way that's transparent to API consumers that currently capture the ssl > module exceptions? The short answer is that users who currently capture the ssl module exceptions need to start catching these exceptions instead when they transition. Cory From wes.turner at gmail.com Sun Jan 22 11:23:59 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 22 Jan 2017 10:23:59 -0600 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> Message-ID: On Sunday, January 22, 2017, Cory Benfield wrote: > > > On 21 Jan 2017, at 13:07, Nick Coghlan > wrote: > > > > > > I agree with Wes that the backwards compatibility guarantees around > > adding new configuration fields should be clarified. > > > > I think it will suffice to say that "new fields are only appended, > > existing fields are never removed, renamed, or reordered". That means > > that: > > > > - tuple unpacking will be forward compatible as long as you use *args at > the end > > - numeric lookup will be forward compatible > > Good idea. Looking at the GnuTLS manual [1], I see a number of potential additional configuration parameters: - session resumption (bool, expiration time) - Trust on first use (SSH-like) - DANE [2] - [1] https://gnutls.org/manual/gnutls.html#Selecting-cryptographic-key-sizes [2] https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities ... IDK about *args (and integer namedtuple field indexing). I also (these days) tend to disagree with items-accessible-as-attributes dicts because dashes and consistency of API. > > This intro section talks about a combined "Context" objection, but the > > implementation has been split into ServerContext and ClientContext. > > > > That split could also use some explanation in the background section of > the PEP. > > Good idea, I?ll expand on that. > > >> Proposed Interface > >> ^^^^^^^^^^^^^^^^^^ > >> > >> The proposed interface for the new module is influenced by the combined > set of > >> limitations of the above implementations. Specifically, as every > implementation > >> *except* OpenSSL requires that each individual cipher be provided, > there is no > >> option but to provide that lowest-common denominator approach. > > > > The second sentence here doesn't match the description of SChannel > > cipher configuration, so I'm not clear on how the proposed interface > > would map to an SChannel backend. > > Yeah, this is a point I?m struggling with internally. > > SChannel?s API is frustratingly limited. The way I see it there are two > options: > > 1. Allow the possibility that SChannel may allow ciphers that others do > not given the same cipher configuration. This is not a good solution, > frankly, because the situation in which this will happen is predominantly > that SChannel will allow modes or key/hash sizes that the other > implementations would forbid, given the same cipher configuration. > 2. Force all other implementations to be as bad as SChannel: that is, to > make it impossible to restrict key sizes and cipher modes on all > implementations because SChannel can?t. GCD, LCD. 3. ciphers__.get(SCHANNEL) OR ciphers > > I don?t really like either of those choices. I *think* 2 is worse than 1, > but I?m not sure about that. People with opinions should really weigh in on > it. > > > This is the one part of the PEP that I think may need to discuss > > transition strategies for libraries and frameworks that currently let > > ssl module exceptions escape to their users: how do they do that in a > > way that's transparent to API consumers that currently capture the ssl > > module exceptions? > > The short answer is that users who currently capture the ssl module > exceptions need to start catching these exceptions instead when they > transition. Are these exceptions redundant? Could they derive from the new TLSError as well as the existing comparable exception? > > Cory > > _______________________________________________ > Security-SIG mailing list > Security-SIG at python.org > https://mail.python.org/mailman/listinfo/security-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Sun Jan 22 11:34:39 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Sun, 22 Jan 2017 16:34:39 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> Message-ID: <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> > On 22 Jan 2017, at 16:23, Wes Turner wrote: > > Looking at the GnuTLS manual [1], I see a number of potential additional configuration parameters: > > - session resumption (bool, expiration time) > - Trust on first use (SSH-like) > - DANE [2] Remember that the goal of this API is not to support every configuration option supported by 1 or more concrete implementations. The goal of this API is to support the common superset of the most-used APIs as needed for TLS. TOFU at the TLS level is not in that scope. DANE is not widely supported. So the only question there is session resumption, which I think may well be in-scope, but probably doesn?t need to go in the API in v1. > ... IDK about *args (and integer namedtuple field indexing). I also (these days) tend to disagree with items-accessible-as-attributes dicts because dashes and consistency of API. Can you elaborate on this? I feel like I?m missing some context. > GCD, LCD. > > 3. ciphers__.get(SCHANNEL) OR ciphers Can you elaborate on this too? > Are these exceptions redundant? Could they derive from the new TLSError as well as the existing comparable exception? This module should be entirely unattached to the ssl module, IMO. This is most important because the ssl module doesn?t exist when Python is not linked against OpenSSL: being unable to define the exceptions in a ?you don?t need OpenSSL module? because Python isn?t linked against OpenSSL seems like a pretty silly problem. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Jan 22 12:01:09 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 22 Jan 2017 11:01:09 -0600 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> Message-ID: On Sunday, January 22, 2017, Cory Benfield wrote: > > On 22 Jan 2017, at 16:23, Wes Turner > wrote: > > Looking at the GnuTLS manual [1], I see a number of potential additional > configuration parameters: > > - session resumption (bool, expiration time) > - Trust on first use (SSH-like) > - DANE [2] > > > Remember that the goal of this API is not to support every configuration > option supported by 1 or more concrete implementations. The goal of this > API is to support the common superset of the most-used APIs as needed for > TLS. TOFU at the TLS level is not in that scope. DANE is not widely > supported. > - OpenSSL 1.1.0 supports DANE - GnuTLS supports DANE - AFAIU, neither SChannel nor Secure Transport yet support DANE So, in order to support e.g. DANE, where would the additional backend-specific configuration occur? > > So the only question there is session resumption, which I think may well > be in-scope, but probably doesn?t need to go in the API in v1. > > ... IDK about *args (and integer namedtuple field indexing). I also (these > days) tend to disagree with items-accessible-as-attributes dicts because > dashes and consistency of API. > > > Can you elaborate on this? I feel like I?m missing some context. > A frozen ordered dict would be preferable because: - Inevitable eventuality of new/additional config parameters - Consistency and readability of API access with [], .__getitem__(), .get() - https://www.python.org/dev/peps/pep-0416/#rejection-notice ... - https://docs.python.org/3.5/library/types.html#types.MappingProxyType NamedTuple is preferable because: - Immutability - Speed - RAM > > GCD, LCD. > > 3. ciphers__.get(SCHANNEL) OR ciphers > > > Can you elaborate on this too? > If defined, backed specific configuration settings could take precedence over platform-neutral defaults. > > Are these exceptions redundant? Could they derive from the new TLSError as > well as the existing comparable exception? > > > This module should be entirely unattached to the ssl module, IMO. This is > most important because the ssl module doesn?t exist when Python is not > linked against OpenSSL: being unable to define the exceptions in a ?you > don?t need OpenSSL module? because Python isn?t linked against OpenSSL > seems like a pretty silly problem. > Good call. > > Cory > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Sun Jan 22 12:14:14 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Sun, 22 Jan 2017 17:14:14 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> Message-ID: <15B89519-41E1-4DC9-A030-521C80AB8E6F@lukasa.co.uk> > On 22 Jan 2017, at 17:01, Wes Turner wrote: > > - OpenSSL 1.1.0 supports DANE > - GnuTLS supports DANE > - AFAIU, neither SChannel nor Secure Transport yet support DANE > > So, in order to support e.g. DANE, where would the additional backend-specific configuration occur? This would be up to the individual specific backends. They are allowed to extend the API provided here in whatever manner they see fit. > A frozen ordered dict would be preferable because: > > - Inevitable eventuality of new/additional config parameters > - Consistency and readability of API access with [], .__getitem__(), .get() > - https://www.python.org/dev/peps/pep-0416/#rejection-notice ... > - https://docs.python.org/3.5/library/types.html#types.MappingProxyType > > NamedTuple is preferable because: > > - Immutability > - Speed > - RAM So immutable types are key, and a truly-frozen ordered dict would be fine from that perspective. However, the key reason I want to use a namedtuple instead of a dict is to discourage ad-hoc extension of the configuration type (with the side benefit of reducing the risk of typo-based configuration and implementation errors. Essentially, the dictionary approach encourages adding ad-hoc keys into the configuration, which *reduces* auditability and encourages the proliferation of essentially implementation-specific keys. > If defined, backed specific configuration settings could take precedence over platform-neutral defaults. So, that?s definitionally true. What?s very hard is coming up with a general way to express those backend-specific configuration settings, which is why I?m not really trying. In fact, it would probably be possible to argue that doing that is impossible. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at cheimes.de Sun Jan 22 12:16:18 2017 From: christian at cheimes.de (Christian Heimes) Date: Sun, 22 Jan 2017 18:16:18 +0100 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> Message-ID: On 2017-01-22 18:01, Wes Turner wrote: > > > On Sunday, January 22, 2017, Cory Benfield > wrote: > > >> On 22 Jan 2017, at 16:23, Wes Turner > > wrote: >> >> Looking at the GnuTLS manual [1], I see a number of potential >> additional configuration parameters: >> >> - session resumption (bool, expiration time) >> - Trust on first use (SSH-like) >> - DANE [2] > > Remember that the goal of this API is not to support every > configuration option supported by 1 or more concrete > implementations. The goal of this API is to support the common > superset of the most-used APIs as needed for TLS. TOFU at the TLS > level is not in that scope. DANE is not widely supported. > > > - OpenSSL 1.1.0 supports DANE > - GnuTLS supports DANE > - AFAIU, neither SChannel nor Secure Transport yet support DANE > > So, in order to support e.g. DANE, where would the additional > backend-specific configuration occur? DANE is irrelevant for PKI and suffers from the same issue as OCSP requests. Melinda is working on a IETF standard for DANE stapling. For TLS API 2.0 we can talk about OCSP stapling, EV and CT. For the first iteration, any advanced feature is out of scope. Please remember, all features have to be implemented for at least four wrappers (Python ssl, cryptography for PyPy, SChannel and SecureTransport). Let's not get ahead of ourselves. Christian From ethan at stoneleaf.us Sun Jan 22 13:11:41 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 22 Jan 2017 10:11:41 -0800 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> Message-ID: <5884F5DD.4090303@stoneleaf.us> On 01/22/2017 04:18 AM, Cory Benfield wrote: > On 21 Jan 2017, at 13:07, Nick Coghlan wrote: >>> Proposed Interface >>> ^^^^^^^^^^^^^^^^^^ >>> >>> The proposed interface for the new module is influenced by the combined set of >>> limitations of the above implementations. Specifically, as every implementation >>> *except* OpenSSL requires that each individual cipher be provided, there is no >>> option but to provide that lowest-common denominator approach. >> >> The second sentence here doesn't match the description of SChannel >> cipher configuration, so I'm not clear on how the proposed interface >> would map to an SChannel backend. > > Yeah, this is a point I?m struggling with internally. > > SChannel?s API is frustratingly limited. The way I see it there are two options: > > 1. Allow the possibility that SChannel may allow ciphers that others do not > given the same cipher configuration. This is not a good solution, frankly, > because the situation in which this will happen is predominantly that SChannel > will allow modes or key/hash sizes that the other implementations would forbid, > given the same cipher configuration. > > 2. Force all other implementations to be as bad as SChannel: that is, to make it > impossible to restrict key sizes and cipher modes on all implementations because > SChannel can?t. > > I don?t really like either of those choices. I *think* 2 is worse than 1, but I?m > not sure about that. People with opinions should really weigh in on it. I am mostly woefully ignorant of these things, but I think option 1 is far preferable to option 2 -- especially if we get to choose which back-end will be used. Even if we can't, I don't think crippling the good actors is the way forward. As long as SChannel does not allow / ignores restrictions can we not issue a warning (probably with the logging module) saying such? -- ~Ethan~ From wes.turner at gmail.com Sun Jan 22 13:55:39 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 22 Jan 2017 12:55:39 -0600 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <15B89519-41E1-4DC9-A030-521C80AB8E6F@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> <15B89519-41E1-4DC9-A030-521C80AB8E6F@lukasa.co.uk> Message-ID: On Sunday, January 22, 2017, Cory Benfield wrote: > > On 22 Jan 2017, at 17:01, Wes Turner > wrote: > > - OpenSSL 1.1.0 supports DANE > - GnuTLS supports DANE > - AFAIU, neither SChannel nor Secure Transport yet support DANE > > So, in order to support e.g. DANE, where would the additional > backend-specific configuration occur? > > > This would be up to the individual specific backends. They are allowed to > extend the API provided here in whatever manner they see fit. > > A frozen ordered dict would be preferable because: > > - Inevitable eventuality of new/additional config parameters > - Consistency and readability of API access with [], .__getitem__(), .get() > - https://www.python.org/dev/peps/pep-0416/#rejection-notice ... > - https://docs.python.org/3.5/library/types.html#types.MappingProxyType > > NamedTuple is preferable because: > > - Immutability > - Speed > - RAM > > > So immutable types are key, and a truly-frozen ordered dict would be fine > from that perspective. > > However, the key reason I want to use a namedtuple instead of a dict is to > discourage ad-hoc extension of the configuration type (with the side > benefit of reducing the risk of typo-based configuration and implementation > errors. Essentially, the dictionary approach encourages adding ad-hoc keys > into the configuration, which *reduces* auditability and encourages the > proliferation of essentially implementation-specific keys. > So then there are two ways to preserve the centralized configuration (with a configuration object): 1. extra_config, a TLSConfiguration parameter pointing to e.g. a (mutable) List or an OrderedDict of additional parameters 2. create an additional configuration object for additional configuration parameters KeyErrors and AttributeErrors are indeed useful for catching configuration errors (in additional to the policy validation opportunity that a configuration object provides) > If defined, backed specific configuration settings could take precedence > over platform-neutral defaults. > > > So, that?s definitionally true. What?s very hard is coming up with a > general way to express those backend-specific configuration settings, which > is why I?m not really trying. In fact, it would probably be possible to > argue that doing that is impossible. > An Ordered of these backend-specific configuration parameters - while maybe not standardizable - could be helpful: cfg.backend_settings[BACKEND_ENUM] = Ordered the() One use case is applications which don't/won't define any actual config code for their app; they just want it to work everywhere (which may mean working with each platform's native TLS library). There's then an external config to be read into TLSConfiguration (and whatever else for backend-specific configuration parameters) > > Cory > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Mon Jan 23 04:27:50 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 23 Jan 2017 09:27:50 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> <15B89519-41E1-4DC9-A030-521C80AB8E6F@lukasa.co.uk> Message-ID: > On 22 Jan 2017, at 18:55, Wes Turner wrote: > > So then there are two ways to preserve the centralized configuration (with a configuration object): > > 1. extra_config, a TLSConfiguration parameter pointing to e.g. a (mutable) List or an OrderedDict of additional parameters > 2. create an additional configuration object for additional configuration parameters Immutable types that contain user-modifyable mutable types are not immutable. ;) They are immutable *references*. I?m shooting for actual immutability here, which is why most of the API surface that accepted lists has been changed to require tuples instead. That means only (2) could work. It may be that the sensible idea is to say that the _BaseContext constructor should take one config object and then a Mapping of implementation-specific-constant to implementation-specific-config. But I don?t know that we need to enshrine that in the API: concrete implementations that need to do this can simply accept those as extra arguments without violating the abstract API. > One use case is applications which don't/won't define any actual config code for their app; they just want it to work everywhere (which may mean working with each platform's native TLS library). There's then an external config to be read into TLSConfiguration (and whatever else for backend-specific configuration parameters) Sure, but how do you choose which implementation to use? For example, Python-on-macOS may find that there are backends available for SecureTransport, OpenSSL, and GnuTLS (say). There is no general correct answer to ?what should I use here?: the answer is up to application developers. The best we could do is provide a hook that says ?what does Python prefer?, and we may well have to do that eventually if we want to include stdlib support for other backends (so that httplib and friends can DTRT). However, we don?t need to put it into this API right now, and I don?t think there?s a good way to do it generically that doesn?t require someone, somewhere to make a choice. My argument is that whomever is choosing should also actually supply the Context as the method to indicate their choice. For example, Requests would have a function that attempts to import a whole bunch of TLS backends and selects the Context it would prefer to use on a given platform (allowing users to supply just TLSConfiguration if they?re that way inclined), while also allowing users to supply initialized Context objects if they want to. Cory From cory at lukasa.co.uk Mon Jan 23 04:28:34 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 23 Jan 2017 09:28:34 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <5884F5DD.4090303@stoneleaf.us> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <5884F5DD.4090303@stoneleaf.us> Message-ID: <23E820EC-4454-4027-95F0-FF6EE7CA64CA@lukasa.co.uk> > On 22 Jan 2017, at 18:11, Ethan Furman wrote: > > As long as SChannel does not allow / ignores restrictions can we not issue > a warning (probably with the logging module) saying such? That?s a matter for the concrete implementation. If the Python standard library grows an SChannel implementation then certainly it can. Cory From njs at pobox.com Mon Jan 23 20:59:25 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 23 Jan 2017 17:59:25 -0800 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <36AFC91B-7D47-4DA4-966A-926185B5B0B2@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <36AFC91B-7D47-4DA4-966A-926185B5B0B2@lukasa.co.uk> Message-ID: On Sun, Jan 22, 2017 at 4:01 AM, Cory Benfield wrote: > >> On 20 Jan 2017, at 23:00, Nathaniel Smith wrote: >> >> - given that for next protocol negotiation we have to accept arbitrary >> bytestrings and the primary value of the NextProtocol class is to >> protect against typos, I wonder if it would simplify things slightly >> to make the attributes of this class just *be* bytestrings. I.e. what >> you have now but without the inheritance from enum. > > While we *could*, I don?t think the value-add of doing this is very high. Basically the only thing it simplifies is the type declaration, and I don?t know that it?s worth doing that. ;) Well, and the code that receives the values, which currently has to handle both enums and bytestrings. Agreed it's not a big deal, it just seems like the value the enum is adding is all negative. >> - what object types *do* you expect to be passed to wrap_buffers? I >> was assuming bytearrays, but the text at the bottom suggests >> file-likes? > > Yeah, good question. I was assuming file-likes as well, but I don?t see any reason we couldn?t also do bytearrays. What do you think the advantage of that is? I've just been used to using bytearrays everywhere for things like socket receive buffers. I find them much more convenient to work with than BytesIO b/c you can do things like pass them directly to send/recv/parsing functions, and they have nice properties like O(1) deletion from the front. It doesn't matter so much for this case where we're just shuffling bytes to and from a socket in a lump, but that's why I was assuming bytearrays without thinking about it :-). I guess h2's data_to_send and data_received don't involve any file-like objects either? >> It isn't quite clear to me right now how this kind of API should look >> with your proposal. Obviously they can't just take a SSLConfiguration >> object, because there's no way to look at an SSLConfiguration object >> and figure out what backend is in use (even though in general a given >> SSLConfiguration is only usable with a specific backend, because >> backends provide the Cert type etc). They could take a >> ServerContext/ClientContext, I guess? But it would be nice if there >> were some way to say "give me the default configuration, using >> SChannel". Or to write a function that sets up an SSLConfiguration >> while being generic over backends, so like it takes the backend as an >> argument and then uses that backend's cert type etc. > > Yeah, I was assuming that they?d take a ClientContext/ServerContext object, rather than a configuration plus an instruction on which backend to use. It?s not clear to me that a function that setups up a configuration while generic over backends is actually a meaningful thing to have: are there really going to be users who are insistent on a specific TLS configuration but don?t care what concrete implementation is going to be used, such that their libraries have to pick it for them? I think it's more likely the other way around -- users need to override the default backend, but don't want to specify all the details? And any library that does build configuration should be able to do it using generic code so long as it's sticking to generic features? requests.get(..., backend=WeirdEmbeddedTLSLib) -> "I need to use this lib, but requests should pick the ciphers/TLS version/etc., I'll just screw it up" web-server.cfg: listen-port = 443 [ssl] enabled = True backend = SChannel certificate = foo.pem Like, in the PEP as currently written, it goes to great efforts to make sure that generic code can be used to select cipher suites, but there's no way for a web server to load in certificate given a PEM filename, except via having a hard-coded table of all possible backends? The PEP says that each backend should somehow somewhere provide a concrete subclass of Certificate, but to find it you need to consult the backend-specific docs. This is really a usability point that's mostly orthogonal to the semantics/design discussion and should be pretty straightforward to sort out, so it might make sense to defer worrying about it until the other stuff's more settled. -n -- Nathaniel J. Smith -- https://vorpus.org From ethan at stoneleaf.us Mon Jan 23 21:16:01 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 23 Jan 2017 18:16:01 -0800 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <36AFC91B-7D47-4DA4-966A-926185B5B0B2@lukasa.co.uk> Message-ID: <5886B8E1.2060507@stoneleaf.us> On 01/23/2017 05:59 PM, Nathaniel Smith wrote: > On Sun, Jan 22, 2017 at 4:01 AM, Cory Benfield wrote: >> On 20 Jan 2017, at 23:00, Nathaniel Smith wrote: >>> - given that for next protocol negotiation we have to accept arbitrary >>> bytestrings and the primary value of the NextProtocol class is to >>> protect against typos, I wonder if it would simplify things slightly >>> to make the attributes of this class just *be* bytestrings. I.e. what >>> you have now but without the inheritance from enum. >> >> While we *could*, I don?t think the value-add of doing this is very high. >> Basically the only thing it simplifies is the type declaration, and I >> don?t know that it?s worth doing that. ;) > > Well, and the code that receives the values, which currently has to > handle both enums and bytestrings. Agreed it's not a big deal, it just > seems like the value the enum is adding is all negative. Enum can be mixed with other types: --- 8< ---------------------------------------- from enum import Enum class NextProtocol(bytes, Enum): H2 = b'h2' H2C = b'h2c' HTTP1 = b'http/1.1' WEBRTC = b'webrtc' C_WEBRTC = b'c-webrtc' FTP = b'ftp' STUN = b'stun.nat-discovery' TURN = b'stun.turn' print(NextProtocol.STUN) # NextProtocol.STUN print(isinstance(NextProtocol.STUN, bytes)) # True --- 8< ---------------------------------------- -- ~Ethan~ From cory at lukasa.co.uk Tue Jan 24 04:06:44 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Tue, 24 Jan 2017 09:06:44 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <36AFC91B-7D47-4DA4-966A-926185B5B0B2@lukasa.co.uk> Message-ID: <061E42C3-FAFC-464A-A628-9B1F99137128@lukasa.co.uk> > On 24 Jan 2017, at 01:59, Nathaniel Smith wrote: > > I've just been used to using bytearrays everywhere for things like > socket receive buffers. I find them much more convenient to work with > than BytesIO b/c you can do things like pass them directly to > send/recv/parsing functions, and they have nice properties like O(1) > deletion from the front. It doesn't matter so much for this case where > we're just shuffling bytes to and from a socket in a lump, but that's > why I was assuming bytearrays without thinking about it :-). I guess > h2's data_to_send and data_received don't involve any file-like > objects either? Yup, that?s very correct. Ok, let?s update to include bytearray (really it should be any buffer type). > Like, in the PEP as currently written, it goes to great efforts to > make sure that generic code can be used to select cipher suites, but > there's no way for a web server to load in certificate given a PEM > filename, except via having a hard-coded table of all possible > backends? The PEP says that each backend should somehow somewhere > provide a concrete subclass of Certificate, but to find it you need to > consult the backend-specific docs. > > This is really a usability point that's mostly orthogonal to the > semantics/design discussion and should be pretty straightforward to > sort out, so it might make sense to defer worrying about it until the > other stuff's more settled. Yeah, this is a good catch. I had missed the problems around Certificate and TrustStore objects. Hrm. Let me have a think about it and revise the PEP. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Wed Jan 25 05:51:55 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 25 Jan 2017 10:51:55 +0000 Subject: [Security-sig] Unified TLS API for Python: Draft 3 Message-ID: <71D39E83-8D52-40DE-B7F0-485235239276@lukasa.co.uk> All, Once again, thanks for your feedback: it?s extremely helpful to be able to have a fairly short cycle time with dedicated review. I think I?ve addressed several of the concerns raised before. The third draft of this document is below. Cory ? PEP: XXX Title: TLS Abstract Base Classes Version: $Revision$ Last-Modified: $Date$ Author: Cory Benfield Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 17-Oct-2016 Python-Version: 3.7 Post-History: 30-Aug-2002 Abstract ======== This PEP would define a standard TLS interface in the form of a collection of abstract base classes. This interface would allow Python implementations and third-party libraries to provide bindings to TLS libraries other than OpenSSL that can be used by tools that expect the interface provided by the Python standard library, with the goal of reducing the dependence of the Python ecosystem on OpenSSL. Rationale ========= In the 21st century it has become increasingly clear that robust and user-friendly TLS support is an extremely important part of the ecosystem of any popular programming language. For most of its lifetime, this role in the Python ecosystem has primarily been served by the `ssl module`_, which provides a Python API to the `OpenSSL library`_. Because the ``ssl`` module is distributed with the Python standard library, it has become the overwhelmingly most-popular method for handling TLS in Python. An extraordinary majority of Python libraries, both in the standard library and on the Python Package Index, rely on the ``ssl`` module for their TLS connectivity. Unfortunately, the preeminence of the ``ssl`` module has had a number of unforeseen side-effects that have had the effect of tying the entire Python ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even in situations where it may provide a worse user experience than alternative TLS implementations, which imposes a cognitive burden and makes it hard to provide "platform-native" experiences. Problems -------- The fact that the ``ssl`` module is build into the standard library has meant that all standard-library Python networking libraries are entirely reliant on the OpenSSL that the Python implementation has been linked against. This leads to the following issues: * It is difficult to take advantage of new, higher-security TLS without recompiling Python to get a new OpenSSL. While there are third-party bindings to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format that the standard library understands, forcing projects that want to use them to maintain substantial compatibility layers. * For Windows distributions of Python, they need to be shipped with a copy of OpenSSL. This puts the CPython development team in the position of being OpenSSL redistributors, potentially needing to ship security updates to the Windows Python distributions when OpenSSL vulnerabilities are released. * For macOS distributions of Python, they need either to be shipped with a copy of OpenSSL or linked against the system OpenSSL library. Apple has formally deprecated linking against the system OpenSSL library, and even if they had not, that library version has been unsupported by upstream for nearly one year as of the time of writing. The CPython development team has started shipping newer OpenSSLs with the Python available from python.org, but this has the same problem as with Windows. * Many systems, including but not limited to Windows and macOS, do not make their system certificate stores available to OpenSSL. This forces users to either obtain their trust roots from elsewhere (e.g. `certifi`_) or to attempt to export their system trust stores in some form. Relying on `certifi`_ is less than ideal, as most system administrators do not expect to receive security-critical software updates from PyPI. Additionally, it is not easy to extend the `certifi`_ trust bundle to include custom roots, or to centrally manage trust using the `certifi`_ model. Even in situations where the system certificate stores are made available to OpenSSL in some form, the experience is still sub-standard, as OpenSSL will perform different validation checks than the platform-native TLS implementation. This can lead to users experiencing different behaviour on their browsers or other platform-native tools than they experience in Python, with little or no recourse to resolve the problem. * Users may wish to integrate with TLS libraries other than OpenSSL for many other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), or because OpenSSL is simply too large and unweildy for the platform (e.g. for embedded Python). Those users are left with the requirement to use third-party networking libraries that can interact with their preferred TLS library or to shim their preferred library into the OpenSSL-specific ``ssl`` module API. Additionally, the ``ssl`` module as implemented today limits the ability of CPython itself to add support for alternative TLS backends, or remove OpenSSL support entirely, should either of these become necessary or useful. The ``ssl`` module exposes too many OpenSSL-specific function calls and features to easily map to an alternative TLS backend. Proposal ======== This PEP proposes to introduce a few new Abstract Base Classes in Python 3.7 to provide TLS functionality that is not so strongly tied to OpenSSL. It also proposes to update standard library modules to use only the interface exposed by these abstract base classes wherever possible. There are three goals here: 1. To provide a common API surface for both core and third-party developers to target their TLS implementations to. This allows TLS developers to provide interfaces that can be used by most Python code, and allows network developers to have an interface that they can target that will work with a wide range of TLS implementations. 2. To provide an API that has few or no OpenSSL-specific concepts leak through. The ``ssl`` module today has a number of warts caused by leaking OpenSSL concepts through to the API: the new ABCs would remove those specific concepts. 3. To provide a path for the core development team to make OpenSSL one of many possible TLS backends, rather than requiring that it be present on a system in order for Python to have TLS support. The proposed interface is laid out below. Abstract Base Classes --------------------- There are several interfaces that require standardisation. Those interfaces are: 1. Configuring TLS, currently implemented by the `SSLContext`_ class in the ``ssl`` module. 2. Wrapping a socket object, currently implemented by the `SSLSocket`_ class in the ``ssl`` module. 3. Providing an in-memory buffer for doing in-memory encryption or decryption with no actual I/O (necessary for asynchronous I/O models), currently implemented by the `SSLObject`_ class in the ``ssl`` module. 4. Applying TLS configuration to the wrapping objects in (2) and (3). Currently this is also implemented by the `SSLContext`_ class in the ``ssl`` module. 5. Specifying TLS cipher suites. There is currently no code for doing this in the standard library: instead, the standard library uses OpenSSL cipher suite strings. 6. Specifying application-layer protocols that can be negotiated during the TLS handshake. 7. Specifying TLS versions. 8. Reporting errors to the caller, currently implemented by the `SSLError`_ class in the ``ssl`` module. 9. Specifying certificates to load, either as client or server certificates. 10. Specifying which trust database should be used to validate certificates presented by a remote peer. 11. Finding a way to get hold of these interfaces at run time. While it is technically possible to define (2) in terms of (3), for the sake of simplicity it is easier to define these as two separate ABCs. Implementations are of course free to implement the concrete subclasses however they see fit. Obviously, (5) doesn't require an abstract base class: instead, it requires a richer API for configuring supported cipher suites that can be easily updated with supported cipher suites for different implementations. (9) is a thorny problem, becuase in an ideal world the private keys associated with these certificates would never end up in-memory in the Python process (that is, the TLS library would collaborate with a Hardware Security Module (HSM) to provide the private key in such a way that it cannot be extracted from process memory). Thus, we need to provide an extensible model of providing certificates that allows concrete implementations the ability to provide this higher level of security, while also allowing a lower bar for those implementations that cannot. This lower bar would be the same as the status quo: that is, the certificate may be loaded from an in-memory buffer or from a file on disk. (10) also represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. Some implementations have specific trust store formats that only they can use (such as the OpenSSL CA directory format that is created by ``c_rehash``), and others may not allow you to specify a trust store that does not include their default trust store. For this reason, we need to provide a model that assumes very little about the form that trust stores take. The "Trust Store" section below goes into more detail about how this is achieved. Finally, this API will split the responsibilities currently assumed by the `SSLContext`_ object: specifically, the responsibility for holding and managing configuration and the responsibility for using that configuration to build wrapper objects. This is necessarily primarily for supporting functionality like Server Name Indication (SNI). In OpenSSL (and thus in the ``ssl`` module), the server has the ability to modify the TLS configuration in response to the client telling the server what hostname it is trying to reach. This is mostly used to change certificate chain so as to present the correct TLS certificate chain for the given hostname. The specific mechanism by which this is done is by returning a new `SSLContext`_ object with the appropriate configuration. This is not a model that maps well to other TLS implementations. Instead, we need to make it possible to provide a return value from the SNI callback that can be used to indicate what configuration changes should be made. This means providing an object that can hold TLS configuration. This object needs to be applied to specific TLSWrappedBuffer, and TLSWrappedSocket objects. For this reason, we split the responsibility of `SSLContext`_ into two separate objects. The ``TLSConfiguration`` object is an object that acts as container for TLS configuration: the ``ClientContext`` and ``ServerContext`` objects are objects that are instantiated with a ``TLSConfiguration`` object. Both objects would be immutable. Configuration ~~~~~~~~~~~~~ The ``TLSConfiguration`` concrete class defines an object that can hold and manage TLS configuration. The goals of this class are as follows: 1. To provide a method of specifying TLS configuration that avoids the risk of errors in typing (this excludes the use of a simple dictionary). 2. To provide an object that can be safely compared to other configuration objects to detect changes in TLS configuration, for use with the SNI callback. This class is not an ABC, primarily because it is not expected to have implementation-specific behaviour. The responsibility for transforming a ``TLSConfiguration`` object into a useful set of configuration for a given TLS implementation belongs to the Context objects discussed below. This class has one other notable property: it is immutable. This is a desirable trait for a few reasons. The most important one is that it allows these objects to be used as dictionary keys, which is potentially extremely valuable for certain TLS backends and their SNI configuration. On top of this, it frees implementations from needing to worry about their configuration objects being changed under their feet, which allows them to avoid needing to carefully synchronize changes between their concrete data structures and the configuration object. This object is extendable: that is, future releases of Python may add configuration fields to this object as they become useful. For backwards-compatibility purposes, new fields are only appended to this object. Existing fields will never be removed, renamed, or reordered. The ``TLSConfiguration`` object would be defined by the following code:: ServerNameCallback = Callable[[TLSBufferObject, Optional[str], TLSConfiguration], Any] _configuration_fields = [ 'validate_certificates', 'certificate_chain', 'ciphers', 'inner_protocols', 'lowest_supported_version', 'highest_supported_version', 'trust_store', 'sni_callback', ] _DEFAULT_VALUE = object() class TLSConfiguration(namedtuple('TLSConfiguration', _configuration_fields)): """ An immutable TLS Configuration object. This object has the following properties: :param validate_certificates bool: Whether to validate the TLS certificates. This switch operates at a very broad scope: either validation is enabled, in which case all forms of validation are performed including hostname validation if possible, or validation is disabled, in which case no validation is performed. Not all backends support having their certificate validation disabled. If a backend does not support having their certificate validation disabled, attempting to set this property to ``False`` will throw a ``TLSError`` when this object is passed into a context object. :param certificate_chain Tuple[Tuple[Certificate],PrivateKey]: The certificate, intermediate certificate, and the corresponding private key for the leaf certificate. These certificates will be offered to the remote peer during the handshake if required. The first Certificate in the list must be the leaf certificate. All subsequent certificates will be offered as intermediate additional certificates. :param ciphers Tuple[CipherSuite]: The available ciphers for TLS connections created with this configuration, in priority order. :param inner_protocols Tuple[Union[NextProtocol, bytes]]: Protocols that connections created with this configuration should advertise as supported during the TLS handshake. These may be advertised using either or both of ALPN or NPN. This list of protocols should be ordered by preference. :param lowest_supported_version TLSVersion: The minimum version of TLS that should be allowed on TLS connections using this configuration. :param highest_supported_version TLSVersion: The maximum version of TLS that should be allowed on TLS connections using this configuration. :param trust_store TrustStore: The trust store that connections using this configuration will use to validate certificates. :param sni_callback Optional[ServerNameCallback]: A callback function that will be called after the TLS Client Hello handshake message has been received by the TLS server when the TLS client specifies a server name indication. Only one callback can be set per ``TLSConfiguration``. If the ``sni_callback`` is ``None`` then the callback is disabled. If the ``TLSConfiguration`` is used for a ``ClientContext`` then this setting will be ignored. The ``callback`` function will be called with three arguments: the first will be the ``TLSBufferObject`` for the connection; the second will be a string that represents the server name that the client is intending to communicate (or ``None`` if the TLS Client Hello does not contain a server name); and the third argument will be the original ``Context``. The server name argument will be the IDNA *decoded* server name. The ``callback`` must return a ``TLSConfiguration`` to allow negotiation to continue. Other return values signal errors. Attempting to control what error is signaled by the underlying TLS implementation is not specified in this API, but is up to the concrete implementation to handle. The Context will do its best to apply the ``TLSConfiguration`` changes from its original configuration to the incoming connection. This will usually include changing the certificate chain, but may also include changes to allowable ciphers or any other configuration settings. """ __slots__ = () def __new__(cls, validate_certificates=None: Optional[bool], certificate_chain=None: Optional[Tuple[Tuple[Certificate], PrivateKey]], ciphers=None: Optional[Tuple[CipherSuite]], inner_protocols=None: Optional[Tuple[Union[NextProtocol, bytes]]], lowest_supported_version=None: Optional[TLSVersion], highest_supported_version=None: Optional[TLSVersion], trust_store=None: Optional[TrustStore], sni_callback=None: Optional[ServerNameCallback]): if validate_certificates is None: validate_certificates = True if ciphers is None: ciphers = DEFAULT_CIPHER_LIST if inner_protocols is None: inner_protocols = [] if lowest_supported_version is None: lowest_supported_version = TLSVersion.TLSv1 if highest_supported_version is None: highest_supported_version = TLSVersion.MAXIMUM_SUPPORTED return super().__new__( cls, validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) def update(self, validate_certificates=_DEFAULT_VALUE, certificate_chain=_DEFAULT_VALUE, ciphers=_DEFAULT_VALUE, inner_protocols=_DEFAULT_VALUE, lowest_supported_version=_DEFAULT_VALUE, highest_supported_version=_DEFAULT_VALUE, trust_store=_DEFAULT_VALUE, sni_callback=_DEFAULT_VALUE): """ Create a new ``TLSConfiguration``, overriding some of the settings on the original configuration with the new settings. """ if validate_certificates is _DEFAULT_VALUE: validate_certificates = self.validate_certificates if certificate_chain is _DEFAULT_VALUE: certificate_chain = self.certificate_chain if ciphers is _DEFAULT_VALUE: ciphers = self.ciphers if inner_protocols is _DEFAULT_VALUE: inner_protocols = self.inner_protocols if lowest_supported_version is _DEFAULT_VALUE: lowest_supported_version = self.lowest_supported_version if highest_supported_version is _DEFAULT_VALUE: highest_supported_version = self.highest_supported_version if trust_store is _DEFAULT_VALUE: trust_store = self.trust_store if sni_callback is _DEFAULT_VALUE: sni_callback = self.sni_callback return self.__class__( validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) Context ~~~~~~~ We define two Context abstract base classes. These ABCs define objects that allow configuration of TLS to be applied to specific connections. They can be thought of as factories for ``TLSWrappedSocket`` and ``TLSWrappedBuffer`` objects. Unlike the current ``ssl`` module, we provide two context classes instead of one. Specifically, we provide the ``ClientContext`` and ``ServerContext`` classes. This simplifies the APIs (for example, there is no sense in the server providing the ``server_hostname`` parameter to ``ssl.SSLContext.wrap_socket``, but because there is only one context class that parameter is still available), and ensures that implementations know as early as possible which side of a TLS connection they will serve. Additionally, it allows implementations to opt-out of one or either side of the connection. For example, SChannel on macOS is not really intended for server use and has an enormous amount of functionality missing for server-side use. This would allow SChannel implementations to simply not define a concrete subclass of ``ServerContext`` to signal their lack of support. As much as possible implementers should aim to make these classes immutable: that is, they should prefer not to allow users to mutate their internal state directly, instead preferring to create new contexts from new TLSConfiguration objects. Obviously, the ABCs cannot enforce this constraint, and so they do not attempt to. The ``Context`` abstract base class has the following class definition:: TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] class _BaseContext(metaclass=ABCMeta): @abstractmethod def __init__(self, configuration: TLSConfiguration): """ Create a new context object from a given TLS configuration. """ @property @abstractmethod def configuration(self) -> TLSConfiguration: """ Returns the TLS configuration that was used to create the context. """ class ClientContext(_BaseContext): @abstractmethod def wrap_socket(self, socket: socket.socket, server_hostname: Optional[str], auto_handshake=True: bool) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``server_hostname`` specifies the hostname of the service which we are connecting to. This allows a single server to host multiple SSL-based services with distinct certificates, quite similarly to HTTP virtual hosts. This is also used to validate the TLS certificate for the given hostname. If hostname validation is not desired, then pass ``None`` for this parameter. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.connect()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any, server_hostname: Optional[str]) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. The buffer objects must be either file objects or objects that implement the buffer protocol. The ``server_hostname`` parameter has the same meaning as in ``wrap_socket``. """ class ServerContext(_BaseContext): @abstractmethod def wrap_socket(self, socket: socket.socket, auto_handshake=True: bool) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The parameter ``auto_handshake`` specifies whether to do the SSL handshake automatically after doing a ``socket.accept()``, or whether the application program will call it explicitly, by invoking the ``TLSWrappedSocket.do_handshake()`` method. Calling ``TLSWrappedSocket.do_handshake()`` explicitly gives the program control over the blocking behavior of the socket I/O involved in the handshake. """ @abstractmethod def wrap_buffers(self, incoming: Any, outgoing: Any) -> TLSWrappedBuffer: """ Wrap a pair of buffer objects (``incoming`` and ``outgoing``) to create an in-memory stream for TLS. The SSL routines will read data from ``incoming`` and decrypt it, and write encrypted data to ``outgoing``. The buffer objects must be either file objects or objects that implement the buffer protocol. """ Socket ~~~~~~ The socket-wrapper ABC will be defined by the ``TLSWrappedSocket`` ABC, which has the following definition:: class TLSWrappedSocket(metaclass=ABCMeta): # The various socket methods all must be implemented. Their definitions # have been elided from this class defintion in the PEP because they # aren't instructive. @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[CipherSuite]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ @abstractmethod def unwrap(self) -> socket.socket: """ Cleanly terminate the TLS connection on this wrapped socket. Once called, this ``TLSWrappedSocket`` can no longer be used to transmit data. Returns the socket that was wrapped with TLS. """ Buffer ~~~~~~ The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, which has the following definition:: class TLSWrappedBuffer(metaclass=ABCMeta): @abstractmethod def read(self, amt=None: int) -> bytes: """ Read up to ``amt`` bytes of data from the input buffer and return the result as a ``bytes`` instance. If ``amt`` is ``None``, will attempt to read until either EOF is reached or until further attempts to read would raise either ``WantReadError`` or ``WantWriteError``. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``read()`` can also cause write operations. """ @abstractmethod def readinto(self, buffer: Any, amt=None: int) -> int: """ Read up to ``amt`` bytes of data from the input buffer into ``buffer``, which must be an object that implements the buffer protocol. Returns the number of bytes read. If ``amt`` is ``None``, will attempt to read until either EOF is reached or until further attempts to read would raise either ``WantReadError`` or ``WantWriteError``, or until the buffer is full. Raises ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``readinto()`` can also cause write operations. """ @abstractmethod def write(self, buf: Any) -> int: """ Write ``buf`` in encrypted form to the output buffer and return the number of bytes written. The ``buf`` argument must be an object supporting the buffer interface. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. As at any time a re-negotiation is possible, a call to ``write()`` can also cause read operations. """ @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[CipherSuite]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. """ @abstractmethod def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ @abstractmethod def shutdown(self) -> None: """ Performs a clean TLS shut down. This should generally be used whenever possible to signal to the remote peer that the content is finished. """ Cipher Suites ~~~~~~~~~~~~~ Supporting cipher suites in a truly library-agnostic fashion is a remarkably difficult undertaking. Different TLS implementations often have *radically* different APIs for specifying cipher suites, but more problematically these APIs frequently differ in capability as well as in style. Some examples are shown below: OpenSSL ^^^^^^^ OpenSSL uses a well-known cipher string format. This format has been adopted as a configuration language by most products that use OpenSSL, including Python. This format is relatively easy to read, but has a number of downsides: it is a string, which makes it remarkably easy to provide bad inputs; it lacks much detailed validation, meaning that it is possible to configure OpenSSL in a way that doesn't allow it to negotiate any cipher at all; and it allows specifying cipher suites in a number of different ways that make it tricky to parse. The biggest problem with this format is that there is no formal specification for it, meaning that the only way to parse a given string the way OpenSSL would is to get OpenSSL to parse it. OpenSSL's cipher strings can look like this:: 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5' This string demonstrates some of the complexity of the OpenSSL format. For example, it is possible for one entry to specify multiple cipher suites: the entry ``ECDH+AESGCM`` means "all ciphers suites that include both elliptic-curve Diffie-Hellman key exchange and AES in Galois Counter Mode". More explicitly, that will expand to four cipher suites:: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" That makes parsing a complete OpenSSL cipher string extremely tricky. Add to the fact that there are other meta-characters, such as "!" (exclude all cipher suites that match this criterion, even if they would otherwise be included: "!MD5" means that no cipher suites using the MD5 hash algorithm should be included), "-" (exclude matching ciphers if they were already included, but allow them to be re-added later if they get included again), and "+" (include the matching ciphers, but place them at the end of the list), and you get an *extremely* complex format to parse. On top of this complexity it should be noted that the actual result depends on the OpenSSL version, as an OpenSSL cipher string is valid so long as it contains at least one cipher that OpenSSL recognises. OpenSSL also uses different names for its ciphers than the names used in the relevant specifications. See the manual page for ``ciphers(1)`` for more details. The actual API inside OpenSSL for the cipher string is simple:: char *cipher_list = ; int rc = SSL_CTX_set_cipher_list(context, cipher_list); This means that any format that is used by this module must be able to be converted to an OpenSSL cipher string for use with OpenSSL. SecureTransport ^^^^^^^^^^^^^^^ SecureTransport is the macOS system TLS library. This library is substantially more restricted than OpenSSL in many ways, as it has a much more restricted class of users. One of these substantial restrictions is in controlling supported cipher suites. Ciphers in SecureTransport are represented by a C ``enum``. This enum has one entry per cipher suite, with no aggregate entries, meaning that it is not possible to reproduce the meaning of an OpenSSL cipher string like "ECDH+AESGCM" without hand-coding which categories each enum member falls into. However, the names of most of the enum members are in line with the formal names of the cipher suites: that is, the cipher suite that OpenSSL calls "ECDHE-ECDSA-AES256-GCM-SHA384" is called "TLS_ECDHE_ECDHSA_WITH_AES_256_GCM_SHA384" in SecureTransport. The API for configuring cipher suites inside SecureTransport is simple:: SSLCipherSuite ciphers[] = {TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ...}; OSStatus status = SSLSetEnabledCiphers(context, ciphers, sizeof(cphers)); SChannel ^^^^^^^^ SChannel is the Windows system TLS library. SChannel has extremely restrictive support for controlling available TLS cipher suites, and additionally adopts a third method of expressing what TLS cipher suites are supported. Specifically, SChannel defines a set of ``ALG_ID`` constants (C unsigned ints). Each of these constants does not refer to an entire cipher suite, but instead an individual algorithm. Some examples are ``CALG_3DES`` and ``CALG_AES_256``, which refer to the bulk encryption algorithm used in a cipher suite, ``CALG_DH_EPHEM`` and ``CALG_RSA_KEYX`` which refer to part of the key exchange algorithm used in a cipher suite, ``CALG_SHA1`` and ``CALG_MD5`` which refer to the message authentication code used in a cipher suite, and ``CALG_ECDSA`` and ``CALG_RSA_SIGN`` which refer to the signing portions of the key exchange algorithm. This can be thought of as the half of OpenSSL's functionality that SecureTransport doesn't have: SecureTransport only allows specifying exact cipher suites, while SChannel only allows specifying *parts* of the cipher suite, while OpenSSL allows both. Determining which cipher suites are allowed on a given connection is done by providing a pointer to an array of these ``ALG_ID`` constants. This means that any suitable API must allow the Python code to determine which ``ALG_ID`` constants must be provided. Proposed Interface ^^^^^^^^^^^^^^^^^^ The proposed interface for the new module is influenced by the combined set of limitations of the above implementations. Specifically, as every implementation *except* OpenSSL requires that each individual cipher be provided, there is no option but to provide that lowest-common denominator approach. The simplest approach is to provide an enumerated type that includes all of the cipher suites defined for TLS. The values of the enum members will be their two-octet cipher identifier as used in the TLS handshake, stored as a tuple of integers. The names of the enum members will be their IANA-registered cipher suite names. Rather than populate this enum by hand, it is likely that we'll define a script that can build it from Christian Heimes' `tlsdb JSON file`_ (warning: large file). This also opens up the possibility of extending the API with additional querying function, such as determining which TLS versions support which ciphers, if that functionality is found to be useful or necessary. If users find this approach to be onerous, a future extension to this API can provide helpers that can reintroduce OpenSSL's aggregation functionality. Because this enum would be enormous, the entire enum is not provided here. Instead, a small sample of entries is provided to give a flavor of how it will appear. :: class CipherSuite(Enum): ... TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA = (0xC0, 0x12) ... TLS_ECDHE_ECDSA_WITH_AES_128_CCM = (0xC0, 0xAC) ... TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 = (0xC0, 0x2B) ... Protocol Negotiation ~~~~~~~~~~~~~~~~~~~~ Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 handshake. While NPN and ALPN are, at their fundamental level, built on top of bytestrings, string-based APIs are frequently problematic as they allow for errors in typing that can be hard to detect. For this reason, this module would define a type that protocol negotiation implementations can pass and be passed. This type would wrap a bytestring to allow for aliases for well-known protocols. This allows us to avoid the problems inherent in typos for well-known protocols, while allowing the full extensibility of the protocol negotiation layer if needed by letting users pass byte strings directly. :: class NextProtocol(Enum): H2 = b'h2' H2C = b'h2c' HTTP1 = b'http/1.1' WEBRTC = b'webrtc' C_WEBRTC = b'c-webrtc' FTP = b'ftp' STUN = b'stun.nat-discovery' TURN = b'stun.turn' TLS Versions ~~~~~~~~~~~~ It is often useful to be able to restrict the versions of TLS you're willing to support. There are many security advantages in refusing to use old versions of TLS, and some misbehaving servers will mishandle TLS clients advertising support for newer versions. The following enumerated type can be used to gate TLS versions. Forward-looking applications should almost never set a maximum TLS version unless they absolutely must, as a TLS backend that is newer than the Python that uses it may support TLS versions that are not in this enumerated type. Additionally, this enumerated type defines two additional flags that can always be used to request either the lowest or highest TLS version supported by an implementation. :: class TLSVersion(Enum): MINIMUM_SUPPORTED SSLv2 SSLv3 TLSv1 TLSv1_1 TLSv1_2 TLSv1_3 MAXIMUM_SUPPORTED Errors ~~~~~~ This module would define three base classes for use with error handling. Unlike many of the the other classes defined here, these classes are not abstract, as they have no behaviour. They exist simply to signal certain common behaviours. Backends should subclass these exceptions in their own packages, but needn't define any behaviour for them. In general, concrete implementations should subclass these exceptions rather than throw them directly. This makes it moderately easier to determine which concrete TLS implementation is in use during debugging of unexpected errors. However, this is not mandatory. The definitions of the errors are below:: class TLSError(Exception): """ The base exception for all TLS related errors from any backend. Catching this error should be sufficient to catch *all* TLS errors, regardless of what backend is used. """ class WantWriteError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is written to the network, or until the output buffer is drained. """ class WantReadError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is read from the network, or until more data is available in the input buffer. """ Certificates ~~~~~~~~~~~~ This module would define an abstract X509 certificate class. This class would have almost no behaviour, as the goal of this module is not to provide all possible relevant cryptographic functionality that could be provided by X509 certificates. Instead, all we need is the ability to signal the source of a certificate to a concrete implementation. For that reason, this certificate implementation defines only constructors. In essence, the certificate object in this module could be as abstract as a handle that can be used to locate a specific certificate. Concrete implementations may choose to provide alternative constructors, e.g. to load certificates from HSMs. If a common interface emerges for doing this, this module may be updated to provide a standard constructor for this use-case as well. Concrete implementations should aim to have Certificate objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. :: class Certificate(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes) -> Certificate: """ Creates a Certificate object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN CERTIFICATE" and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, AnyStr]) -> Certificate: """ Creates a Certificate object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. """ Private Keys ~~~~~~~~~~~~ This module would define an abstract private key class. Much like the Certificate class, this class has almost no behaviour in order to give as much freedom as possible to the concrete implementations to treat keys carefully. This class has all the caveats of the ``Certificate`` class. :: class PrivateKey(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes, password=None: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: """ Creates a PrivateKey object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN", the key type, and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. The key may additionally be encrypted. If it is, the ``password`` argument can be used to decrypt the key. The ``password`` argument may be a function to call to get the password for decrypting the private key. It will only be called if the private key is encrypted and a password is necessary. It will be called with no arguments, and it should return either bytes or bytearray containing the password. Alternatively a bytes, or bytearray value may be supplied directly as the password argument. It will be ignored if the private key is not encrypted and no password is needed. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, bytes, str], password=None: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]) -> PrivateKey: """ Creates a PrivateKey object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. The ``password`` parameter behaves exactly as the equivalent parameter on ``from_buffer``. """ Trust Store ~~~~~~~~~~~ As discussed above, loading a trust store represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. For this reason, we need to provide a model that assumes very little about the form that trust stores take. This problem is the same as the one that the Certificate and PrivateKey types need to solve. For this reason, we use the exact same model, by creating an opaque type that can encapsulate the various means that TLS backends may open a trust store. A given TLS implementation is not required to implement all of the constructors. However, it is strongly recommended that a given TLS implementation provide the ``system`` constructor if at all possible, as this is the most common validation trust store that is used. Concrete implementations may also add their own constructors. Concrete implementations should aim to have TrustStore objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. :: class TrustStore(metaclass=ABCMeta): @abstractclassmethod def system(cls) -> TrustStore: """ Returns a TrustStore object that represents the system trust database. """ @abstractclassmethod def from_pem_file(cls, path: Union[pathlib.Path, bytes, str]) -> TrustStore: """ Initializes a trust store from a single file full of PEMs. """ Runtime Access ~~~~~~~~~~~~~~ A not-uncommon use case for library users is to want to allow the library to control the TLS configuration, but to want to select what backend is in use. For example, users of Requests may want to be able to select between OpenSSL or a platform-native solution on Windows and macOS, or between OpenSSL and NSS on some Linux platforms. These users, however, may not care about exactly how their TLS configuration is done. This poses a problem: given an arbitrary concrete implementation, how can a library work out how to load certificates into the trust store? There are two options: either all concrete implementations can be required to fit into a specific naming scheme, or we can provide an API that makes it possible to grab these objects. This PEP proposes that we use the second approach. This grants the greatest freedom to concrete implementations to structure their code as they see fit, requiring only that they provide a single object that has the appropriate properties in place. Users can then pass this "backend" object to libraries that support it, and those libraries can take care of configuring and using the concrete implementation. All concrete implementations must provide a method of obtaining a ``Backend`` object. The ``Backend`` object can be a global singleton or can be created by a callable if there is an advantage in doing that. The ``Backend`` object has the following definition:: Backend = namedtuple( 'Backend', ['client_context', 'server_context', 'certificate', 'private_key', 'trust_store'] ) Each of the properties must provide the concrete implementation of the relevant ABC. This ensures that code like this will work for any backend:: trust_store = backend.trust_store.system() Changes to the Standard Library =============================== The portions of the standard library that interact with TLS should be revised to use these ABCs. This will allow them to function with other TLS backends. This includes the following modules: - asyncio - ftplib - http.client - imaplib - nntplib - poplib - smtplib Future ====== Major future TLS features may require revisions of these ABCs. These revisions should be made cautiously: many backends may not be able to move forward swiftly, and will be invalidated by changes in these ABCs. This is acceptable, but wherever possible features that are specific to individual implementations should not be added to the ABCs. The ABCs should restrict themselves to high-level descriptions of IETF-specified features. However, well-justified extensions to this API absolutely should be made. The focus of this API is to provide a unifying lowest-common-denominator configuration option for the Python community. TLS is not a static target, and as TLS evolves so must this API. References ========== .. _ssl module: https://docs.python.org/3/library/ssl.html .. _OpenSSL Library: https://www.openssl.org/ .. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ .. _certifi: https://pypi.org/project/certifi/ .. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext .. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket .. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject .. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError .. _MSDN articles: https://msdn.microsoft.com/en-us/library/windows/desktop/mt490158(v=vs.85).aspx .. _tlsdb JSON file: https://github.com/tiran/tlsdb/blob/master/tlsdb.json Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From ethan at stoneleaf.us Wed Jan 25 21:24:33 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 Jan 2017 18:24:33 -0800 Subject: [Security-sig] email & phones Message-ID: <58895DE1.1060706@stoneleaf.us> Question for everyone -- particularly those that reply to messages using their phones (an honest question, I'm not trying to be a jerk): Is it very difficult to trim quoted text that you are not replying to? On Python List I'm a stickler for kicking back messages where large parts of a previous email (usually the whole thing) are quoted for apparently no good reason as I feel such replies seriously drop the signal-to-noise ratio. However, this list is very low traffic, so if trimming quoted text is a huge pain when using phone email I could stop worrying about that. Thoughts? -- ~Ethan~ From donald at stufft.io Wed Jan 25 21:26:31 2017 From: donald at stufft.io (Donald Stufft) Date: Wed, 25 Jan 2017 21:26:31 -0500 Subject: [Security-sig] email & phones In-Reply-To: <58895DE1.1060706@stoneleaf.us> References: <58895DE1.1060706@stoneleaf.us> Message-ID: <1CAB6D9C-9E6E-4573-A17C-118AC07CCA62@stufft.io> > On Jan 25, 2017, at 9:24 PM, Ethan Furman wrote: > > Is it very difficult to trim quoted text that you are not replying to? It is generally more difficult than on the computer because accurately highlighting a single line of text is more annoying, and going through and clicking through an email to delete and type inline replies is a lot more annoying and finicky as well. I also think a large part of it is just that a lot of popular email clients (particularly gmail) completely hide the post you?re responding to in the reply with a little show/hide widget. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed Jan 25 21:34:52 2017 From: steve.dower at python.org (Steve Dower) Date: Wed, 25 Jan 2017 18:34:52 -0800 Subject: [Security-sig] email & phones In-Reply-To: <58895DE1.1060706@stoneleaf.us> References: <58895DE1.1060706@stoneleaf.us> Message-ID: For me, yes it's impossible. I literally cannot modify the post below the line, and copy-pasting the relevant quotes is far from convenient. In general, I try to avoid replying from my phone if any of my four PCs are close, but often it's the difference between replying and never-replying. My signature is my apology, and also redirecting blame where it belongs :) (I'll get a new phone when I need one - so far it fulfils my needs just fine.) Top-posted from my Windows Phone -----Original Message----- From: "Ethan Furman" Sent: ?1/?25/?2017 18:25 To: "Python Security SIG" Subject: [Security-sig] email & phones Question for everyone -- particularly those that reply to messages using their phones (an honest question, I'm not trying to be a jerk): Is it very difficult to trim quoted text that you are not replying to? On Python List I'm a stickler for kicking back messages where large parts of a previous email (usually the whole thing) are quoted for apparently no good reason as I feel such replies seriously drop the signal-to-noise ratio. However, this list is very low traffic, so if trimming quoted text is a huge pain when using phone email I could stop worrying about that. Thoughts? -- ~Ethan~ _______________________________________________ Security-SIG mailing list Security-SIG at python.org https://mail.python.org/mailman/listinfo/security-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Jan 25 21:53:59 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 Jan 2017 18:53:59 -0800 Subject: [Security-sig] Unified TLS API for Python: Draft 3 In-Reply-To: <71D39E83-8D52-40DE-B7F0-485235239276@lukasa.co.uk> References: <71D39E83-8D52-40DE-B7F0-485235239276@lukasa.co.uk> Message-ID: <588964C7.3050305@stoneleaf.us> Originally from Steve Dower (with my apologies): ----------------------------------------------- Looks good to me, but I wonder why we need to define all the algorithms in the PEP (by reference)? Could they also use a similar model to certificates, where the implementation provides a constructor that takes a string (in a format defined here, e.g. "OpenSSL style") and does the best it can to return something it will know how to use later? It involves some trust, but I honestly don't see a world where we end up with implementations deliberately trying to be incompatible with each other (which would seem to be the only reason to define the full enum ahead of time). Steve Dower (extracted from a top-posted Windows phone email ;) From ethan at stoneleaf.us Wed Jan 25 23:35:49 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 25 Jan 2017 20:35:49 -0800 Subject: [Security-sig] email & phones In-Reply-To: <58895DE1.1060706@stoneleaf.us> References: <58895DE1.1060706@stoneleaf.us> Message-ID: <58897CA5.7030604@stoneleaf.us> Thanks for the responses. I'll stop blocking large phone replies and investigate being able to strip extra content below the "top posted from my windows phone" line. And thanks for your patience -- as you can probably tell, I don't do email from my phone so had no reference points. -- ~Ethan~ From ncoghlan at gmail.com Thu Jan 26 02:49:53 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Jan 2017 08:49:53 +0100 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> Message-ID: On 22 January 2017 at 17:34, Cory Benfield wrote: > This module should be entirely unattached to the ssl module, IMO. This is > most important because the ssl module doesn?t exist when Python is not > linked against OpenSSL: being unable to define the exceptions in a ?you > don?t need OpenSSL module? because Python isn?t linked against OpenSSL seems > like a pretty silly problem. The way Python handles inheritance means that we have a fair bit of flexibility in how we handle the exception catching transition. To be clear, the specific case where I'm interested in improving the migration strategy is the one where: - a networking helper library inside a project or app handles ssl API calls on behalf of its clients - the helper library wants to migrate to use the tls API instead - the only aspect of the ssl API exposed directly to users of the helper library is the exceptions, with everything else being handled by the library Option 1: No assistance is provided, so either all SSL exception catching code in clients of the helper library *must* be updated to also catch the corresponding new TLS exceptions in order to be agnostic regarding the use of the old ssl API or the new tls one, or else the library has to define its own wrapper exceptions that inherit from both the old ssl exceptions and the new tls ones Option 2: ssl.SSLError, ssl.SSLWantReadError, and ssl.SSLWantWriteError are redefined as subclasses of tls.TLSError, tls.WantReadError, tls.WantWriteError respectively Option 3: the tls API defines tls.LegacySSLError, tls.LegacyWantReadError, tls.LegacyWantWriteError as aliases for ssl.SSLError, ssl.SSLWantReadError, and ssl.SSLWantWriteError *if* the latter are defined, and otherwise defines them as new exception types Option 4: tls.TLSError, tls.WantReadError, tls.WantWriteError are defined as inheriting from ssl.SSLError, ssl.SSLWantReadError, and ssl.SSLWantWriteError *if* the latter are defined Option 5: as with Option 4, but the "ssl" module is also changed such that it *always* defines at least ssl.SSLError, ssl.SSLWantReadError, and ssl.SSLWantWriteError (and perhaps some of the other APIs that can be emulated atop the new tls abstraction), even if OpenSSL itself is unavailable I really don't like Option 1, as it just pushes the decision on how to deal with this migration to library designers, rather than coming up with a standardised approach that minimises the collective hassle for the overall ecosystem Option 2 isn't particularly appealing either - it still requires that all code catching these exceptions be modified in order to cope with underlying libraries migrating from using the legacy ssl API to the updated tls API, and it also isn't particularly amenable to being backported to earlier Python versions Option 3 starts being slightly more helpful to API consumers, but still doesn't really help library authors hide the backend migration for their API clients Option 4 by contrast means that library authors can still hide the migration to the new backend APIs even if they previously allowed the ssl module exceptions to escape as part of their public API - any clients catching those exceptions will still catch the new exceptions, and will only require changing in order to support running in environments that don't provide an OpenSSL backend at all Option 5 would cover even that last case: legacy API consumers that only relied on being able to catch the legacy exceptions would tolerate the use of non-OpenSSL backends even in environments where OpenSSL itself wasn't available Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From christian at cheimes.de Thu Jan 26 03:50:10 2017 From: christian at cheimes.de (Christian Heimes) Date: Thu, 26 Jan 2017 09:50:10 +0100 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> Message-ID: <4ee361b9-fe42-10e8-462d-efc0dbc1982a@cheimes.de> On 2017-01-26 08:49, Nick Coghlan wrote: [...] > Option 5: as with Option 4, but the "ssl" module is also changed such > that it *always* defines at least ssl.SSLError, ssl.SSLWantReadError, > and ssl.SSLWantWriteError (and perhaps some of the other APIs that can > be emulated atop the new tls abstraction), even if OpenSSL itself is > unavailable [...] > Option 5 would cover even that last case: legacy API consumers that > only relied on being able to catch the legacy exceptions would > tolerate the use of non-OpenSSL backends even in environments where > OpenSSL itself wasn't available Hi Nick, I'm a bit worried that option 5 is wasting resources and/or has unwanted side effects. Import of ssl is costly because it also loads and initializes OpenSSL. It's an unnecessary burden for applications that do not wish to use OpenSSL (macOS SecureTransport, Windows SChannel) at all or not the bundled OpenSSL version (static builds of cryptography). How about we move the exceptions and the base class for the TLSWrappedSocket to the `socket` module instead? In CPython the exception would live in _socket and get exported as PyCapsule. The socket module provides class TLSError(OSError): """socket.TLSError""" class TLWantWriteError(TLSError): """socket.TLSWantWriteError""" class TLWantReadError(TLSError): """socket.TLSWantReadError""" class AbstractSocket(meta=abc.ABCMeta) """socket.AbstractSockt""" The tls module provides: import socket from socket import TLSError, TLSWantReadError, TLSWantWriteError class TLSWrappedSocket(socket.AbstractSocket): pass Christian From cory at lukasa.co.uk Thu Jan 26 04:18:11 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 26 Jan 2017 09:18:11 +0000 Subject: [Security-sig] Unified TLS API for Python: Draft 3 In-Reply-To: References: <71D39E83-8D52-40DE-B7F0-485235239276@lukasa.co.uk> Message-ID: > On 26 Jan 2017, at 01:33, Steve Dower wrote: > > Looks good to me, but I wonder why we need to define all the algorithms in the PEP (by reference)? Could they also use a similar model to certificates, where the implementation provides a constructor that takes a string (in a format defined here, e.g. "OpenSSL style") and does the best it can to return something it will know how to use later? It involves some trust, but I honestly don't see a world where we end up with implementations deliberately trying to be incompatible with each other (which would seem to be the only reason to define the full enum ahead of time). There?s a thread running through this PEP which I?d call a rejection of ?stringly-typed? APIs. I have never liked OpenSSL?s cipher string format: it?s a constant source of footguns where a given cipher string provides an extremely non-deterministic result. Because OpenSSL needs to tolerate the possibility that you are referring to ciphers that are only present in older or newer versions of OpenSSL, there is little error checking done on a cipher string. You can see this by running the openssl ciphers command on two very similar cipher strings: openssl ciphers "ECDHE+AESGCM:RSA+AESGCM" ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256 vs openssl ciphers "ECDHE+AESGCM:RSA+AESGMC" ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256 Note that the typo in the second string wipes out a whole bunch of cipher suites, but OpenSSL doesn?t flag this as a problem because it doesn?t know whether AESGMC is a thing that might appear in the future or have appeared in the past. The only way to actually error check this is to compare to the hard-coded list of what cipher suites you expect, except of course at that point you may as well have just written the full list of cipher suites out or, more aptly, used a typed API. OpenSSL only errors out on a cipher string if it leads to no ciphers being supported, which is an extremely unlikely outcome for any project taking rigorous control of their cipher string. The other problem here is that we need to define a grammar of how to translate the cipher string into a list of actual ciphers. This grammar needs to be forward-looking, which is a problem that OpenSSL is running directly into as we speak[0]. For this reason I?m inclined to lean towards the more verbose approach of just writing down what all of the cipher suites are in an enum. That way, it gets much easier to validate what?s going on. There?s still no requirement to actually support them all: an implementation is allowed to quietly ignore any cipher suites it doesn?t support. But that can no longer happen due to typos, because typos now cause AttributeErrors at runtime in a way that is very obvious and clear. Does that make sense? Cory [0]: This is a digression, but worth harping on about. OpenSSL?s cipher suite names have traditionally been of the form ?key exchange-signing algo-stream cipher-mode-MAC?. However, OpenSSL has allowed some of those to be unspecified and to mean their defaults (e.g. AES128-GCM-SHA256 actually means RSA-RSA-AES128-GCM-SHA256). This has caused them problems with TLSv1.3, which has defined all new cipher suites that no longer include their key exchange mechanism (e.g. TLS_AES_128_GCM_SHA256 is now a complete cipher string). Because of OpenSSL?s weird defaulty grammar, right now their cipher string assumes that this new cipher uses RSA key exchange, even though it absolutely does not. All of this is a very long form way of saying that defining a stable string-to-cipher-suite parser is very hard unless we define it as saying ?just use the actual IANA names for the cipher suites?, and the second we?ve done that we may as well have just defined an enum and called it good. ;) From cory at lukasa.co.uk Thu Jan 26 04:50:56 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 26 Jan 2017 09:50:56 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> Message-ID: <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> > On 26 Jan 2017, at 07:49, Nick Coghlan wrote: > > Option 4: tls.TLSError, tls.WantReadError, tls.WantWriteError are > defined as inheriting from ssl.SSLError, ssl.SSLWantReadError, and > ssl.SSLWantWriteError *if* the latter are defined > > Option 5: as with Option 4, but the "ssl" module is also changed such > that it *always* defines at least ssl.SSLError, ssl.SSLWantReadError, > and ssl.SSLWantWriteError (and perhaps some of the other APIs that can > be emulated atop the new tls abstraction), even if OpenSSL itself is > unavailable Here?s my problem with this: try: socket.recv(8192) except tls.WantWriteError: socket.write(some_buffer) This code does not work with the legacy ssl module, because isinstance(ssl.SSLWantWriteError, tls.WantWriteError) is false. This means that we need to write a shim over the legacy ssl module that wraps *all* API calls, catches all exceptions and then translates them into subclasses of the tls error classes. That seems entirely batty to me. Cory From njs at pobox.com Thu Jan 26 05:49:28 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 26 Jan 2017 02:49:28 -0800 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> Message-ID: On Thu, Jan 26, 2017 at 1:50 AM, Cory Benfield wrote: > >> On 26 Jan 2017, at 07:49, Nick Coghlan wrote: >> >> Option 4: tls.TLSError, tls.WantReadError, tls.WantWriteError are >> defined as inheriting from ssl.SSLError, ssl.SSLWantReadError, and >> ssl.SSLWantWriteError *if* the latter are defined >> >> Option 5: as with Option 4, but the "ssl" module is also changed such >> that it *always* defines at least ssl.SSLError, ssl.SSLWantReadError, >> and ssl.SSLWantWriteError (and perhaps some of the other APIs that can >> be emulated atop the new tls abstraction), even if OpenSSL itself is >> unavailable > > Here?s my problem with this: > > try: > socket.recv(8192) > except tls.WantWriteError: > socket.write(some_buffer) > > This code does not work with the legacy ssl module, because isinstance(ssl.SSLWantWriteError, tls.WantWriteError) is false. This means that we need to write a shim over the legacy ssl module that wraps *all* API calls, catches all exceptions and then translates them into subclasses of the tls error classes. That seems entirely batty to me. It seems like the simplest effective solution to these problems would be for ssl in 3.7 to do ssl.py: from tls import TLSError as SSLError, WantWriteError as SSLWantWriteError, WantReadError as SSLWantReadError and then legacy code that catches SSLWant{Write,Read}Error will be automatically ported forward to the new TLS world. And in the backported version of the tls module for older Pythons, we could have it do the reverse to accomplish a similar effect (at the cost of importing ssl -- but this seems unavoidable in old-Python): tls.py: from ssl import SSLError as TLSError, SSLWantWriteError as WantWriteError, SSLWantReadError as WantReadError There's really no case where it's important to distinguish these, right? -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Thu Jan 26 09:23:45 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Jan 2017 15:23:45 +0100 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> Message-ID: On 26 January 2017 at 10:50, Cory Benfield wrote: > >> On 26 Jan 2017, at 07:49, Nick Coghlan wrote: >> >> Option 4: tls.TLSError, tls.WantReadError, tls.WantWriteError are >> defined as inheriting from ssl.SSLError, ssl.SSLWantReadError, and >> ssl.SSLWantWriteError *if* the latter are defined >> >> Option 5: as with Option 4, but the "ssl" module is also changed such >> that it *always* defines at least ssl.SSLError, ssl.SSLWantReadError, >> and ssl.SSLWantWriteError (and perhaps some of the other APIs that can >> be emulated atop the new tls abstraction), even if OpenSSL itself is >> unavailable > > Here?s my problem with this: > > try: > socket.recv(8192) > except tls.WantWriteError: > socket.write(some_buffer) > > This code does not work with the legacy ssl module, because isinstance(ssl.SSLWantWriteError, tls.WantWriteError) is false. This means that we need to write a shim over the legacy ssl module that wraps *all* API calls, catches all exceptions and then translates them into subclasses of the tls error classes. That seems entirely batty to me. OK, so we have two competing problems here: 1. How do we write *new* API client code that is agnostic to whether or not the security implementation is traditional ssl or a new tls backend? 2. How does a library that exports the ssl exceptions migrate to using a tls backend instead, without having to catch and rewrap tls exceptions in the legacy ones? Meeting both constraints at the same time would require exception *aliases* rather than one set of exceptions inheriting from the other, such that we get: assert tls.TLSError is ssl.SSLError assert tls.WantWriteError is ssl.SSLWantWriteError assert tls.WantReadError is ssl.SSLWantReadError In an ideal world, that could be handled just by having the ssl module import the new tls module and alias the exceptions accordingly. Talking to Christian about it, things might be a little messier in practice due to the way the _ssl/ssl C/Python split works, but there shouldn't be any insurmountable barriers to going down the exception aliasing path in 3.7+. Backports to older versions would still need to contend with these being different exception objects, but one possible way of tackling that would be to inject a dummy ssl module into sys.modules before the regular one had a chance to be imported. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Jan 26 09:27:04 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Jan 2017 15:27:04 +0100 Subject: [Security-sig] email & phones In-Reply-To: <58895DE1.1060706@stoneleaf.us> References: <58895DE1.1060706@stoneleaf.us> Message-ID: On 26 January 2017 at 03:24, Ethan Furman wrote: > However, this list is very low traffic, so if trimming quoted text is a huge > pain when using phone email I could stop worrying about that. In the gmail mobile client, deleting the entire quoted post is straightforward, but trimming it when quoting bits selectively is a PITA. Basically, it's set up on the assumption that all recipients will be using a client that autohides quoted text, and only expands it when requested to do so. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cory at lukasa.co.uk Thu Jan 26 10:22:07 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 26 Jan 2017 15:22:07 +0000 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> Message-ID: > On 26 Jan 2017, at 14:23, Nick Coghlan wrote: > > In an ideal world, that could be handled just by having the ssl module > import the new tls module and alias the exceptions accordingly. > Talking to Christian about it, things might be a little messier in > practice due to the way the _ssl/ssl C/Python split works, but there > shouldn't be any insurmountable barriers to going down the exception > aliasing path in 3.7+. I?d be ok with going down the aliasing route. Is this a concern worth noting in the PEP, do you think? Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at cheimes.de Thu Jan 26 11:06:16 2017 From: christian at cheimes.de (Christian Heimes) Date: Thu, 26 Jan 2017 17:06:16 +0100 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> Message-ID: <45ef45fb-62ba-8dcf-ef36-50ee14c02177@cheimes.de> On 2017-01-26 15:23, Nick Coghlan wrote: > On 26 January 2017 at 10:50, Cory Benfield wrote: >> >>> On 26 Jan 2017, at 07:49, Nick Coghlan wrote: >>> >>> Option 4: tls.TLSError, tls.WantReadError, tls.WantWriteError are >>> defined as inheriting from ssl.SSLError, ssl.SSLWantReadError, and >>> ssl.SSLWantWriteError *if* the latter are defined >>> >>> Option 5: as with Option 4, but the "ssl" module is also changed such >>> that it *always* defines at least ssl.SSLError, ssl.SSLWantReadError, >>> and ssl.SSLWantWriteError (and perhaps some of the other APIs that can >>> be emulated atop the new tls abstraction), even if OpenSSL itself is >>> unavailable >> >> Here?s my problem with this: >> >> try: >> socket.recv(8192) >> except tls.WantWriteError: >> socket.write(some_buffer) >> >> This code does not work with the legacy ssl module, because isinstance(ssl.SSLWantWriteError, tls.WantWriteError) is false. This means that we need to write a shim over the legacy ssl module that wraps *all* API calls, catches all exceptions and then translates them into subclasses of the tls error classes. That seems entirely batty to me. > > OK, so we have two competing problems here: > > 1. How do we write *new* API client code that is agnostic to whether > or not the security implementation is traditional ssl or a new tls > backend? > 2. How does a library that exports the ssl exceptions migrate to using > a tls backend instead, without having to catch and rewrap tls > exceptions in the legacy ones? > > Meeting both constraints at the same time would require exception > *aliases* rather than one set of exceptions inheriting from the other, > such that we get: > > assert tls.TLSError is ssl.SSLError > assert tls.WantWriteError is ssl.SSLWantWriteError > assert tls.WantReadError is ssl.SSLWantReadError > > In an ideal world, that could be handled just by having the ssl module > import the new tls module and alias the exceptions accordingly. > Talking to Christian about it, things might be a little messier in > practice due to the way the _ssl/ssl C/Python split works, but there > shouldn't be any insurmountable barriers to going down the exception > aliasing path in 3.7+. > > Backports to older versions would still need to contend with these > being different exception objects, but one possible way of tackling > that would be to inject a dummy ssl module into sys.modules before the > regular one had a chance to be imported. For technical reasons it is beneficial to define SSL exception in C code. I don't want to force people to load _ssl to avoid the overhead of OpenSSL loading and initialization. After some brain storming with Nick I came up with the idea to define the exceptions in _socket. The _ssl module imports a PyCapsule from _socket any way. I can just piggyback on the PyCapsule. For maximum compatibility with Python 2, the tls module should subclass from socket.error anyway, too. Python 2: >>> import ssl >>> ssl.SSLError.__mro__ (, , , , , , , ) Python 3: >>> import ssl, socket >>> ssl.SSLError.__mro__ (, , , , ) >>> socket.error This import block should (hopefully) cover all bases: try: # CPython 3.7+ from _socket import SSLError as TLSError from _socket import SSLWantWriteError as WantWriteError from _socket import SSLWantReadError as WantReadError except ImportError: # CPython < 3.7 and other implementations try: from ssl import SSLError as TLSError from ssl import SSLWantWriteError as WantWriteError from ssl import SSLWantReadError as WantReadError except ImportError: # ssl module is not available # Python 2: socket.error is a subclass of IOError # Python 3: socket.error is just an alias of OSError import socket class TLSError(socket.error): pass class WantWriteError(TLSError): pass class WantReadError(TLSError): pass Christian From ncoghlan at gmail.com Thu Jan 26 11:14:28 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 26 Jan 2017 17:14:28 +0100 Subject: [Security-sig] Unified TLS API for Python: Round 2 In-Reply-To: References: <54B4B794-E103-42B2-89F5-0A64AD470C94@lukasa.co.uk> <54B8565D-6584-4C89-AC3B-7FDCF2DE5202@lukasa.co.uk> <6635164D-8C87-422D-9CC6-3674EAABCBC1@lukasa.co.uk> <979C0354-A37E-4133-B1F4-590929D10DB6@lukasa.co.uk> Message-ID: On 26 January 2017 at 16:22, Cory Benfield wrote: > > On 26 Jan 2017, at 14:23, Nick Coghlan wrote: > > In an ideal world, that could be handled just by having the ssl module > import the new tls module and alias the exceptions accordingly. > Talking to Christian about it, things might be a little messier in > practice due to the way the _ssl/ssl C/Python split works, but there > shouldn't be any insurmountable barriers to going down the exception > aliasing path in 3.7+. > > I?d be ok with going down the aliasing route. Is this a concern worth noting > in the PEP, do you think? I think so, as the aliasing means that: - new code can just catch the tls exceptions and automatically catch the old ssl exceptions as well - old code that just catches the old exceptions will also catch the new exceptions, so helper library authors don't need to worry about catching the new exceptions and re-raising them as the old ones I don't think you need to explain the technical details of how the aliasing would work though - that really is an implementation detail (although you may want to provide a reference to Christian's email that spells it out in full). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Thu Jan 26 16:17:43 2017 From: donald at stufft.io (Donald Stufft) Date: Thu, 26 Jan 2017 16:17:43 -0500 Subject: [Security-sig] Unified TLS API for Python: Draft 3 In-Reply-To: References: <71D39E83-8D52-40DE-B7F0-485235239276@lukasa.co.uk> Message-ID: <6C5C8936-D70D-4EC3-810A-6C481BF71DDA@stufft.io> > On Jan 26, 2017, at 4:18 AM, Cory Benfield wrote: > > For this reason I?m inclined to lean towards the more verbose approach of just writing down what all of the cipher suites are in an enum. That way, it gets much easier to validate what?s going on. There?s still no requirement to actually support them all: an implementation is allowed to quietly ignore any cipher suites it doesn?t support. But that can no longer happen due to typos, because typos now cause AttributeErrors at runtime in a way that is very obvious and clear. I?d say additionally that given the verbose approach a third party library could provide this OpenSSL like API and be responsible for ?compiling? it down to the actual list of ciphers for input into the verbose API. If one of those got popular and seemed stable enough to add it, we could always add it in later as a higher level API for cipher selection without the backends needing to change anything since the output of such a function would still be a list of all of the desired ciphers which would be the input to the backends. ? Donald Stufft -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Fri Jan 27 04:10:40 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 27 Jan 2017 09:10:40 +0000 Subject: [Security-sig] Unified TLS API for Python: Draft 3 In-Reply-To: <6C5C8936-D70D-4EC3-810A-6C481BF71DDA@stufft.io> References: <71D39E83-8D52-40DE-B7F0-485235239276@lukasa.co.uk> <6C5C8936-D70D-4EC3-810A-6C481BF71DDA@stufft.io> Message-ID: > On 26 Jan 2017, at 21:17, Donald Stufft wrote: > > >> On Jan 26, 2017, at 4:18 AM, Cory Benfield > wrote: >> >> For this reason I?m inclined to lean towards the more verbose approach of just writing down what all of the cipher suites are in an enum. That way, it gets much easier to validate what?s going on. There?s still no requirement to actually support them all: an implementation is allowed to quietly ignore any cipher suites it doesn?t support. But that can no longer happen due to typos, because typos now cause AttributeErrors at runtime in a way that is very obvious and clear. > > > I?d say additionally that given the verbose approach a third party library could provide this OpenSSL like API and be responsible for ?compiling? it down to the actual list of ciphers for input into the verbose API. If one of those got popular and seemed stable enough to add it, we could always add it in later as a higher level API for cipher selection without the backends needing to change anything since the output of such a function would still be a list of all of the desired ciphers which would be the input to the backends. Yup, strongly agreed. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Jan 27 10:30:00 2017 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 27 Jan 2017 09:30:00 -0600 Subject: [Security-sig] Unified TLS API for Python: Draft 3 In-Reply-To: References: <71D39E83-8D52-40DE-B7F0-485235239276@lukasa.co.uk> <6C5C8936-D70D-4EC3-810A-6C481BF71DDA@stufft.io> Message-ID: On Fri, Jan 27, 2017 at 3:10 AM, Cory Benfield wrote: > > On 26 Jan 2017, at 21:17, Donald Stufft wrote: > > > On Jan 26, 2017, at 4:18 AM, Cory Benfield wrote: > > For this reason I?m inclined to lean towards the more verbose approach of > just writing down what all of the cipher suites are in an enum. That way, > it gets much easier to validate what?s going on. There?s still no > requirement to actually support them all: an implementation is allowed to > quietly ignore any cipher suites it doesn?t support. But that can no longer > happen due to typos, because typos now cause AttributeErrors at runtime in > a way that is very obvious and clear. > > > > I?d say additionally that given the verbose approach a third party library > could provide this OpenSSL like API and be responsible for ?compiling? it > down to the actual list of ciphers for input into the verbose API. If one > of those got popular and seemed stable enough to add it, we could always > add it in later as a higher level API for cipher selection without the > backends needing to change anything since the output of such a function > would still be a list of all of the desired ciphers which would be the > input to the backends. > > > Yup, strongly agreed. > https://github.com/tiran/tlsdb/blob/master/tlsdb.py - [ ] ENH: tlsdb.py: add parsers/datasources for {SChannel, SecureTransport} - [x] openssl-master - [x] openssl-1.02 - [x] gnutls-master - [x] nss-tip - [x] mod_nss-master - [x] **iana** - [x] mozilla-server-side - [ ] SChannel - [ ] SecureTransport - [ ] ENH: tlsdb.py: add OpenSSL-workalike lookup method - [ ] BLD: tls.config.__: generate Enums? > > Cory > > _______________________________________________ > Security-SIG mailing list > Security-SIG at python.org > https://mail.python.org/mailman/listinfo/security-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: