From ncoghlan at gmail.com Sun Jul 3 02:30:27 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 2 Jul 2016 23:30:27 -0700 Subject: [Security-sig] Does the buildtime HAVE_GETRANDOM_SYSCALL check actually make sense? In-Reply-To: References: Message-ID: On 26 June 2016 at 02:30, Victor Stinner wrote: > The configure check ensures that constants required to build random.c are > available. We can only run this check at the compilation. I don't want to > maintain hardcoded constants. > > The proper fix is to add getrandom() to the libc: > https://sourceware.org/bugzilla/show_bug.cgi?id=17252 > > But you may have the same issue if you build the lib with "old" header > files. mock (the Fedora build utility, not the Python testing utility) will use the updated header files when building against the newer glibc, so not worrying about this until it's resolved there sounds fine to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Jul 3 02:37:37 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 2 Jul 2016 23:37:37 -0700 Subject: [Security-sig] Does the buildtime HAVE_GETRANDOM_SYSCALL check actually make sense? In-Reply-To: References: Message-ID: On 2 July 2016 at 23:30, Nick Coghlan wrote: > On 26 June 2016 at 02:30, Victor Stinner wrote: >> The configure check ensures that constants required to build random.c are >> available. We can only run this check at the compilation. I don't want to >> maintain hardcoded constants. >> >> The proper fix is to add getrandom() to the libc: >> https://sourceware.org/bugzilla/show_bug.cgi?id=17252 >> >> But you may have the same issue if you build the lib with "old" header >> files. > > mock (the Fedora build utility, not the Python testing utility) will > use the updated header files when building against the newer glibc, so > not worrying about this until it's resolved there sounds fine to me. Huh, I should have caught up on the issue I filed against Fedora first - it appears there's something going wrong with the config scripts, since the syscall *is* available and working on the build servers. Either way, nothing for upstream to worry about until we figure out why the config script is giving the wrong answer (since my original theory of not having access to the syscall turned out to be wrong). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Thu Jul 28 11:11:21 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 28 Jul 2016 17:11:21 +0200 Subject: [Security-sig] How to document changes related to security in Python changelog? In-Reply-To: References: <57695492.6060100@stoneleaf.us> <20160621184016.4ad487a3.barry@wooz.org> Message-ID: Hi, I pushed my change to tag security related changes with a [Security] prefix in Python 3.5.2 changes. I was too lazy to make it for older and more recent changes. 3.5: https://hg.python.org/cpython/rev/a576a34f5386 default: https://hg.python.org/cpython/rev/6a2de662eeb7 Victor 2016-06-27 23:15 GMT+02:00 Victor Stinner : > Ok, I wrote a first patch to mark changes related to security in > Python 3.5.2 changelog: > https://bugs.python.org/issue27404 > > Victor From victor.stinner at gmail.com Thu Jul 28 12:56:45 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 28 Jul 2016 18:56:45 +0200 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) Message-ID: Hi, I moved my os.urandom PEP to the regular GitHub repository so it gets available on python.org: https://www.python.org/dev/peps/pep-0524/ Changes since version 2: * add os.getrandom() * random.Random now always use non-blocking system urandom and so remove the "Never use blocking urandom in the random module" section * add a new "Examples using os.getrandom()" section * various fixes and minor updates Victor PEP: 524 Title: Make os.urandom() blocking on Linux Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-June-2016 Python-Version: 3.6 Abstract ======== Modify ``os.urandom()`` to block on Linux 3.17 and newer until the OS urandom is initialized to increase the security. Add also a new ``os.getrandom()`` function (for Linux and Solaris) to be able to choose how to handle when ``os.urandom()`` is going to block on Linux. The bug ======= Original bug ------------ Python 3.5.0 was enhanced to use the new ``getrandom()`` syscall introduced in Linux 3.17 and Solaris 11.3. The problem is that users started to complain that Python 3.5 blocks at startup on Linux in virtual machines and embedded devices: see issues `#25420 `_ and `#26839 `_. On Linux, ``getrandom(0)`` blocks until the kernel initialized urandom with 128 bits of entropy. The issue #25420 describes a Linux build platform blocking at ``import random``. The issue #26839 describes a short Python script used to compute a MD5 hash, systemd-cron, script called very early in the init process. The system initialization blocks on this script which blocks on ``getrandom(0)`` to initialize Python. The Python initilization requires random bytes to implement a counter-measure against the hash denial-of-service (hash DoS), see: * `Issue #13703: Hash collision security issue `_ * `PEP 456: Secure and interchangeable hash algorithm `_ Importing the ``random`` module creates an instance of ``random.Random``: ``random._inst``. On Python 3.5, random.Random constructor reads 2500 bytes from ``os.urandom()`` to seed a Mersenne Twister RNG (random number generator). Other platforms may be affected by this bug, but in practice, only Linux systems use Python scripts to initialize the system. Status in Python 3.5.2 ---------------------- Python 3.5.2 behaves like Python 2.7 and Python 3.4. If the system urandom is not initialized, the startup does not block, but ``os.urandom()`` can return low-quality entropy (even it is not easily guessable). Use Cases ========= The following use cases are used to help to choose the right compromise between security and practicability. Use Case 1: init script ----------------------- Use a Python 3 script to initialize the system, like systemd-cron. If the script blocks, the system initialize is stuck too. The issue #26839 is a good example of this use case. Use case 1.1: No secret needed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If the init script doesn't have to generate any secure secret, this use case is already handled correctly in Python 3.5.2: Python startup doesn't block on system urandom anymore. Use case 1.2: Secure secret required ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If the init script has to generate a secure secret, there is no safe solution. Falling back to weak entropy is not acceptable, it would reduce the security of the program. Python cannot produce itself secure entropy, it can only wait until system urandom is initialized. But in this use case, the whole system initialization is blocked by this script, so the system fails to boot. The real answer is that the system iniitalization must not be blocked by such script. It is ok to start the script very early at system initialization, but the script may blocked a few seconds until it is able to generate the secret. Reminder: in some cases, the initialization of the system urandom never occurs and so programs waiting for system urandom blocks forever. Use Case 2: Web server ---------------------- Run a Python 3 web server serving web pages using HTTP and HTTPS protocols. The server is started as soon as possible. The first target of the hash DoS attack was web server: it's important that the hash secret cannot be easily guessed by an attacker. If serving a web page needs a secret to create a cookie, create an encryption key, ..., the secret must be created with good entropy: again, it must be hard to guess the secret. A web server requires security. If a choice must be made between security and running the server with weak entropy, security is more important. If there is no good entropy: the server must block or fail with an error. The question is if it makes sense to start a web server on a host before system urandom is initialized. The issues #25420 and #26839 are restricted to the Python startup, not to generate a secret before the system urandom is initialized. Fix system urandom ================== Load entropy from disk at boot ------------------------------- Collecting entropy can take up to several minutes. To accelerate the system initialization, operating systems store entropy on disk at shutdown, and then reload entropy from disk at the boot. If a system collects enough entropy at least once, the system urandom will be initialized quickly, as soon as the entropy is reloaded from disk. Virtual machines ---------------- Virtual machines don't have a direct access to the hardware and so have less sources of entropy than bare metal. A solution is to add a `virtio-rng device `_ to pass entropy from the host to the virtual machine. Embedded devices ---------------- A solution for embedded devices is to plug an hardware RNG. For example, Raspberry Pi have an hardware RNG but it's not used by default. See: `Hardware RNG on Raspberry Pi `_. Denial-of-service when reading random ===================================== Don't use /dev/random but /dev/urandom -------------------------------------- The ``/dev/random`` device should only used for very specific use cases. Reading from ``/dev/random`` on Linux is likely to block. Users don't like when an application blocks longer than 5 seconds to generate a secret. It is only expected for specific cases like generating explicitly an encryption key. When the system has no available entropy, choosing between blocking until entropy is available or falling back on lower quality entropy is a matter of compromise between security and practicability. The choice depends on the use case. On Linux, ``/dev/urandom`` is secure, it should be used instead of ``/dev/random``. See `Myths about /dev/urandom `_ by Thomas H?hn: "Fact: /dev/urandom is the preferred source of cryptographic randomness on UNIX-like systems" getrandom(size, 0) can block forever on Linux --------------------------------------------- The origin of the Python issue #26839 is the `Debian bug report #822431 `_: in fact, ``getrandom(size, 0)`` blocks forever on the virtual machine. The system succeeded to boot because systemd killed the blocked process after 90 seconds. Solutions like `Load entropy from disk at boot`_ reduces the risk of this bug. Rationale ========= On Linux, reading the ``/dev/urandom`` can return "weak" entropy before urandom is fully initialized, before the kernel collected 128 bits of entropy. Linux 3.17 adds a new ``getrandom()`` syscall which allows to block until urandom is initialized. On Python 3.5.2, os.urandom() uses the ``getrandom(size, GRND_NONBLOCK)``, but falls back on reading the non-blocking ``/dev/urandom`` if ``getrandom(size, GRND_NONBLOCK)`` fails with ``EAGAIN``. Security experts promotes ``os.urandom()`` to genereate cryptographic keys because it is implemented with a `Cryptographically secure pseudo-random number generator (CSPRNG) `_. By the way, ``os.urandom()`` is preferred over ``ssl.RAND_bytes()`` for different reasons. This PEP proposes to modify os.urandom() to use ``getrandom()`` in blocking mode to not return weak entropy, but also ensure that Python will not block at startup. Changes ======= Make os.urandom() blocking on Linux ----------------------------------- All changes described in this section are specific to the Linux platform. Changes: * Modify os.urandom() to block until system urandom is initialized: ``os.urandom()`` (C function ``_PyOS_URandom()``) is modified to always call ``getrandom(size, 0)`` (blocking mode) on Linux and Solaris. * Add a new private ``_PyOS_URandom_Nonblocking()`` function: try to call ``getrandom(size, GRND_NONBLOCK)`` on Linux and Solaris, but falls back on reading ``/dev/urandom`` if it fails with ``EAGAIN``. * Initialize hash secret from non-blocking system urandom: ``_PyRandom_Init()`` is modified to call ``_PyOS_URandom_Nonblocking()``. * ``random.Random`` constructor now uses non-blocking system urandom: it is modified to use internally the new ``_PyOS_URandom_Nonblocking()`` function to seed the RNG. Add a new os.getrandom() function --------------------------------- A new ``os.getrandom(size, flags=0)`` function is added: use ``getrandom()`` syscall on Linux and ``getrandom()`` C function on Solaris. The function comes with 2 new flags: * ``os.GRND_RANDOM``: read bytes from ``/dev/random`` rather than reading ``/dev/urandom`` * ``os.GRND_NONBLOCK``: raise a BlockingIOError if ``os.getrandom()`` would block The ``os.getrandom()`` is a thin wrapper on the ``getrandom()`` syscall/C function and so inherit of its behaviour. For example, on Linux, it can return less bytes than requested if the syscall is interrupted by a signal. Examples using os.getrandom() ============================= Best-effort RNG --------------- Example of a portable non-blocking RNG function: try to get random bytes from the OS urandom, or fallback on the random module. :: def best_effort_rng(size): # getrandom() is only available on Linux and Solaris if not hasattr(os, 'getrandom'): return os.urandom(size) result = bytearray() try: # need a loop because getrandom() can return less bytes than # requested for different reasons while size: data = os.getrandom(size, os.GRND_NONBLOCK) result += data size -= len(data) except BlockingIOError: # OS urandom is not initialized yet: # fallback on the Python random module data = bytes(random.randrange(256) for byte in range(size)) result += data return bytes(result) This function *can* block in theory on a platform where ``os.getrandom()`` is not available but ``os.urandom()`` can block. wait_for_system_rng() --------------------- Example of function waiting *timeout* seconds until the OS urandom is initialized on Linux or Solaris:: def wait_for_system_rng(timeout, interval=1.0): if not hasattr(os, 'getrandom'): return deadline = time.monotonic() + timeout while True: try: os.getrandom(1, os.GRND_NONBLOCK) except BlockingIOError: pass else: return if time.monotonic() > deadline: raise Exception('OS urandom not initialized after %s seconds' % timeout) time.sleep(interval) This function is *not* portable. For example, ``os.urandom()`` can block on FreeBSD in theory, at the early stage of the system initialization. Create a best-effort RNG ------------------------ Simpler example to create a non-blocking RNG on Linux: choose between ``Random.SystemRandom`` and ``Random.Random`` depending if ``getrandom(size)`` would block. :: def create_nonblocking_random(): if not hasattr(os, 'getrandom'): return random.Random() try: os.getrandom(1, os.GRND_NONBLOCK) except BlockingIOError: return random.Random() else: return random.SystemRandom() This function is *not* portable. For example, ``random.SystemRandom`` can block on FreeBSD in theory, at the early stage of the system initialization. Alternative =========== Leave os.urandom() unchanged, add os.getrandom() ------------------------------------------------ os.urandom() remains unchanged: never block, but it can return weak entropy if system urandom is not initialized yet. Only add the new ``os.getrandom()`` function (wrapper to the ``getrandom()`` syscall/C function). The ``secrets.token_bytes()`` function should be used to write portable code. The problem with this change is that it expects that users understand well security and know well each platforms. Python has the tradition of hiding "implementation details". For example, ``os.urandom()`` is not a thin wrapper to the ``/dev/urandom`` device: it uses ``CryptGenRandom()`` on Windows, it uses ``getentropy()`` on OpenBSD, it tries ``getrandom()`` on Linux and Solaris or falls back on reading ``/dev/urandom``. Python already uses the best available system RNG depending on the platform. This PEP does not change the API: * ``os.urandom()``, ``random.SystemRandom`` and ``secrets`` for security * ``random`` module (except ``random.SystemRandom``) for all other usages Raise BlockingIOError in os.urandom() ------------------------------------- Proposition ^^^^^^^^^^^ `PEP 522: Allow BlockingIOError in security sensitive APIs on Linux `_. Python should not decide for the developer how to handle `The bug`_: raising immediatly a ``BlockingIOError`` if ``os.urandom()`` is going to block allows developers to choose how to handle this case: * catch the exception and falls back to a non-secure entropy source: read ``/dev/urandom`` on Linux, use the Python ``random`` module (which is not secure at all), use time, use process identifier, etc. * don't catch the error, the whole program fails with this fatal exception More generally, the exception helps to notify when sometimes goes wrong. The application can emit a warning when it starts to wait for ``os.urandom()``. Criticism ^^^^^^^^^ For the use case 2 (web server), falling back on non-secure entropy is not acceptable. The application must handle ``BlockingIOError``: poll ``os.urandom()`` until it completes. Example:: def secret(n=16): try: return os.urandom(n) except BlockingIOError: pass print("Wait for system urandom initialiation: move your " "mouse, use your keyboard, use your disk, ...") while 1: # Avoid busy-loop: sleep 1 ms time.sleep(0.001) try: return os.urandom(n) except BlockingIOError: pass For correctness, all applications which must generate a secure secret must be modified to handle ``BlockingIOError`` even if `The bug`_ is unlikely. The case of applications using ``os.urandom()`` but don't really require security is not well defined. Maybe these applications should not use ``os.urandom()`` at the first place, but always the non-blocking ``random`` module. If ``os.urandom()`` is used for security, we are back to the use case 2 described above: `Use Case 2: Web server`_. If a developer doesn't want to drop ``os.urandom()``, the code should be modified. Example:: def almost_secret(n=16): try: return os.urandom(n) except BlockingIOError: return bytes(random.randrange(256) for byte in range(n)) The question is if `The bug`_ is common enough to require that so many applications have to be modified. Another simpler choice is to refuse to start before the system urandom is initialized:: def secret(n=16): try: return os.urandom(n) except BlockingIOError: print("Fatal error: the system urandom is not initialized") print("Wait a bit, and rerun the program later.") sys.exit(1) Compared to Python 2.7, Python 3.4 and Python 3.5.2 where os.urandom() never blocks nor raise an exception on Linux, such behaviour change can be seen as a major regression. Add an optional block parameter to os.urandom() ----------------------------------------------- See the `issue #27250: Add os.urandom_block() `_. Add an optional block parameter to os.urandom(). The default value may be ``True`` (block by default) or ``False`` (non-blocking). The first technical issue is to implement ``os.urandom(block=False)`` on all platforms. Only Linux 3.17 (and newer) and Solaris 11.3 (and newer) have a well defined non-blocking API (``getrandom(size, GRND_NONBLOCK)``). As `Raise BlockingIOError in os.urandom()`_, it doesn't seem worth it to make the API more complex for a theorical (or at least very rare) use case. As `Leave os.urandom() unchanged, add os.getrandom()`_, the problem is that it makes the API more complex and so more error-prone. Annexes ======= Operating system random functions --------------------------------- ``os.urandom()`` uses the following functions: * `OpenBSD: getentropy() `_ (OpenBSD 5.6) * `Linux: getrandom() `_ (Linux 3.17) -- see also `A system call for random numbers: getrandom() `_ * Solaris: `getentropy() `_, `getrandom() `_ (both need Solaris 11.3) * UNIX, BSD: /dev/urandom, /dev/random * Windows: `CryptGenRandom() `_ (Windows XP) On Linux, commands to get the status of ``/dev/random`` (results are number of bytes):: $ cat /proc/sys/kernel/random/entropy_avail 2850 $ cat /proc/sys/kernel/random/poolsize 4096 Why using os.urandom()? ----------------------- Since ``os.urandom()`` is implemented in the kernel, it doesn't have issues of user-space RNG. For example, it is much harder to get its state. It is usually built on a CSPRNG, so even if its state is "stolen", it is hard to compute previously generated numbers. The kernel has a good knowledge of entropy sources and feed regulary the entropy pool. That's also why ``os.urandom()`` is preferred over ``ssl.RAND_bytes()``. Copyright ========= This document has been placed in the public domain. From ncoghlan at gmail.com Fri Jul 29 07:35:33 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 29 Jul 2016 21:35:33 +1000 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: Message-ID: On 29 July 2016 at 02:56, Victor Stinner wrote: > Hi, > > I moved my os.urandom PEP to the regular GitHub repository so it gets > available on python.org: > https://www.python.org/dev/peps/pep-0524/ > > Changes since version 2: > > * add os.getrandom() I'm wondering if we should take os.getrandom() out of the PEPs, and just agree that adding it as a platform-dependent optional interface in the OS module is a good idea. The reason I've started thinking that way is that one of the capabilities we've long supported is to let people write platform specific "C'ish" code in Python, that then later gets translated to actual C code, either manually, or automatically via Cython (or one of its predecessors). Adding os.getrandom() would make sense in that context, just as it's the reason we keep "getopt" around as a standard library module. The aspect of Larry's original proposal to add os.getrandom that I objected to was adding it *instead of* fixing os.urandom's default behaviour, but I think we've moved past that point of contention now. The competing PEPs would then just be about where we put the cross-platform blocking behaviour: either doing it implicitly in os.urandom, or letting that sometimes throw BlockingIOError, and offering an explicit cross-platform blocking API in the secrets module. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Jul 29 08:54:28 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 29 Jul 2016 14:54:28 +0200 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: Message-ID: Hi, 2016-07-29 13:35 GMT+02:00 Nick Coghlan : > I'm wondering if we should take os.getrandom() out of the PEPs, and > just agree that adding it as a platform-dependent optional interface > in the OS module is a good idea. It's a deliberate choice to add os.getrandom() as part of my PEP. It is a solution for concrete use case: be able to detect when os.urandom() will block and decide how to handle this case (implement "polling"). That's also why I added examples using os.getrandom(). Your PEP 522 contains many examples and use cases to be notified when os.urandom() will block and decide how to handle it (implement polling). That's why I completed my PEP, these use cases look common enough to justify the addition of os.getrandom(). Victor From barry at python.org Fri Jul 29 09:31:55 2016 From: barry at python.org (Barry Warsaw) Date: Fri, 29 Jul 2016 09:31:55 -0400 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: Message-ID: <20160729093155.2fec8274.barry@wooz.org> On Jul 29, 2016, at 09:35 PM, Nick Coghlan wrote: >I'm wondering if we should take os.getrandom() out of the PEPs, and >just agree that adding it as a platform-dependent optional interface >in the OS module is a good idea. I agree that os.getrandom() should be added on platforms that support it regardless of the outcome of the various PEPs, and that it should be a thin layer above the C function. Cheers, -Barry From victor.stinner at gmail.com Fri Jul 29 10:21:17 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 29 Jul 2016 16:21:17 +0200 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: <20160729093155.2fec8274.barry@wooz.org> References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: 2016-07-29 15:31 GMT+02:00 Barry Warsaw : > I agree that os.getrandom() should be added on platforms that support it > regardless of the outcome of the various PEPs, and that it should be a thin > layer above the C function. Yes, in my PEP 524, os.getrandom() is a thin wrapper to getrandom(). Well, "thin" wrapper... An interesting question is if the function should be implemented using a loop or only do exactly one call to getrandom()? Examples: - getrandom() is limited to 1024 bytes on Solaris - getrandom() can return less bytes than requested on Linux if the call is interrupted by a signal. If os.getrandom() only do a single call, the function must be called in a loop at the Python level: result = bytearray() while size: data = os.getrandom(size) result += data size -= len(data) return bytes(result) Currently, _PyOS_URandom() implements the loop at the C level. The implementation of os.getrandom() must respect the PEP 475 (retry syscall on EINTR), so maybe we should implement the loop at the C level. Ok, but imagine that you use the "expensive" GRNG_RANDOM (/dev/random rather than /dev/urandom). A first call returns 4000 bytes of "high quality" random bytes, but user requested 4096 bytes and the second call fails. Is it ok to "drop" the 4000 collected bytes? Entropy is a limited resource... The strict minimum is to implement os.getrandom() with a single call, *but* retry the getrandom() call if it fails with EINTR and the Python signal handler doesn't raise any exception. With this design, we don't drop any collected byte. But os.getrandom() should be used with a loop at the Python level. Victor From victor.stinner at gmail.com Fri Jul 29 10:30:15 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 29 Jul 2016 16:30:15 +0200 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: <20160729093155.2fec8274.barry@wooz.org> References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: 2016-07-29 15:31 GMT+02:00 Barry Warsaw : > I agree that os.getrandom() should be added on platforms that support it > regardless of the outcome of the various PEPs, and that it should be a thin > layer above the C function. According to the size of the discussion, I consider that it's worth it to use a PEP to add os.getrandom(). Exposing getrandom() was proposed in different ways which was also part of the discussion. The fact that getrandom() is directly exposed or not has a major impact on the overall PEP. For example, I don't think that it's worth to add os.getrandom() if the PEP 522 is implemented. The PEP 522 directly gives access to non-blocking system urandom, so os.getrandom(GRNG_NONBLOCK) is useless (it's less portable and more complex to use, see my other email in this thread). The remaining question is if it is woth to expose os.getrandom(GRNG_RANDOM) ... PEP 522 and PEP 524 advice to avoid /dev/random which should really be reserved for a very limited number of application. These applications can already open manually /dev/random, code which would work on all Python versions! Finally, only adding os.getrandom() was also proposed as the complete solution to "the bug": users are expected to move slowly to this new function. Nick explained it: https://www.python.org/dev/peps/pep-0522/#why-propose-this-now So I don't consider that it's tiny detail in the discussion... Victor From ncoghlan at gmail.com Fri Jul 29 11:34:21 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 30 Jul 2016 01:34:21 +1000 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: On 30 July 2016 at 00:30, Victor Stinner wrote: > 2016-07-29 15:31 GMT+02:00 Barry Warsaw : > According to the size of the discussion, I consider that it's worth it > to use a PEP to add os.getrandom(). Exposing getrandom() was proposed > in different ways which was also part of the discussion. > > The fact that getrandom() is directly exposed or not has a major > impact on the overall PEP. > > For example, I don't think that it's worth to add os.getrandom() if > the PEP 522 is implemented. The PEP 522 directly gives access to > non-blocking system urandom, so os.getrandom(GRNG_NONBLOCK) is useless > (it's less portable and more complex to use, see my other email in > this thread). The remaining question is if it is woth to expose > os.getrandom(GRNG_RANDOM) ... PEP 522 and PEP 524 advice to avoid > /dev/random which should really be reserved for a very limited number > of application. These applications can already open manually > /dev/random, code which would work on all Python versions! > > Finally, only adding os.getrandom() was also proposed as the complete > solution to "the bug": users are expected to move slowly to this new > function. Nick explained it: > https://www.python.org/dev/peps/pep-0522/#why-propose-this-now > > So I don't consider that it's tiny detail in the discussion... Right, to elaborate on my point of view a bit further, I'm not entirely happy with the current design in PEP 522, as it doesn't offer: - a way of accessing the system level blocking behaviour on Linux (you have to emulate it with a loop) - a way to directly transcribe getrandom() based code to or from C on modern Linux or Solaris So adding os.getrandom() doesn't necessarily have anything to do with fixing people's code, and instead has to do with providing access to the corresponding kernel capability on platforms that provide it -it's a "What is the os module for?" argument rather than a "How do we encourage good security programming practices?" argument. It also isn't something that can be readily provided via cffi or ctypes, as glibc doesn't offer an API for it yet - you have to call the syscall directly, which I believe is possible via cffi/ctypes, but presumably tough to make robust against platform variations. PEPs 522 and 524 then become a matter of posing the question, "Given the existence of os.getrandom(), how do we fix currently at risk uses of the os.urandom() function?": * tell people to use os.getrandom() if available, os.urandom() otherwise (both PEPs rejected) * make them fail noisily, with a cross-platform way to opt-in to blocking (PEP 522) * make them "just work" in most cases, but potentially block in some situations (PEP 524) So what I'll probably do is update PEP 522 to assume os.getrandom() will be available (with the above rationale), and then see how that changes the rest of the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Jul 29 11:37:49 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 30 Jul 2016 01:37:49 +1000 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: On 30 July 2016 at 00:21, Victor Stinner wrote: > The implementation of os.getrandom() must respect the PEP 475 (retry > syscall on EINTR), so maybe we should implement the loop at the C > level. This was my thought when reading the PEP - I'd be very surprised if I got a short read from os.getrandom() without the GRNG_RANDOM flag. > Ok, but imagine that you use the "expensive" GRNG_RANDOM (/dev/random > rather than /dev/urandom). A first call returns 4000 bytes of "high > quality" random bytes, but user requested 4096 bytes and the second > call fails. I'd say in the non-EINTR case with GRNG_RANDOM it would be reasonable to return a short read. So folks using that flag would need a Python-level loop, but folks using the kernel's CSPRNG wouldn't. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Fri Jul 29 12:52:34 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 29 Jul 2016 18:52:34 +0200 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: 2016-07-29 17:37 GMT+02:00 Nick Coghlan : >> Ok, but imagine that you use the "expensive" GRNG_RANDOM (/dev/random >> rather than /dev/urandom). A first call returns 4000 bytes of "high >> quality" random bytes, but user requested 4096 bytes and the second >> call fails. > > I'd say in the non-EINTR case with GRNG_RANDOM it would be reasonable > to return a short read. So folks using that flag would need a > Python-level loop, but folks using the kernel's CSPRNG wouldn't. Do you mean return the 4000 bytes and *ignores* the error? Victor From barry at python.org Fri Jul 29 14:29:42 2016 From: barry at python.org (Barry Warsaw) Date: Fri, 29 Jul 2016 14:29:42 -0400 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: <20160729142942.3f945e24@anarchist.lan> On Jul 29, 2016, at 04:21 PM, Victor Stinner wrote: >The strict minimum is to implement os.getrandom() with a single call, >*but* retry the getrandom() call if it fails with EINTR and the Python >signal handler doesn't raise any exception. With this design, we don't >drop any collected byte. But os.getrandom() should be used with a loop >at the Python level. Yes, I'd opt for this. I definitely don't think we should be discarding entropy, and I think a Python-level loop should be just fine. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From victor.stinner at gmail.com Fri Jul 29 17:37:47 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 29 Jul 2016 23:37:47 +0200 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: <20160729142942.3f945e24@anarchist.lan> References: <20160729093155.2fec8274.barry@wooz.org> <20160729142942.3f945e24@anarchist.lan> Message-ID: 2016-07-29 20:29 GMT+02:00 Barry Warsaw : >>The strict minimum is to implement os.getrandom() with a single call, >>*but* retry the getrandom() call if it fails with EINTR and the Python >>signal handler doesn't raise any exception. With this design, we don't >>drop any collected byte. But os.getrandom() should be used with a loop >>at the Python level. > > Yes, I'd opt for this. I definitely don't think we should be discarding > entropy, and I think a Python-level loop should be just fine. Ok, fine. I will elaborate this part of my PEP next time I will update it :-) Victor From ncoghlan at gmail.com Sat Jul 30 00:55:04 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 30 Jul 2016 14:55:04 +1000 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: On 30 July 2016 at 02:52, Victor Stinner wrote: > 2016-07-29 17:37 GMT+02:00 Nick Coghlan : >>> Ok, but imagine that you use the "expensive" GRNG_RANDOM (/dev/random >>> rather than /dev/urandom). A first call returns 4000 bytes of "high >>> quality" random bytes, but user requested 4096 bytes and the second >>> call fails. >> >> I'd say in the non-EINTR case with GRNG_RANDOM it would be reasonable >> to return a short read. So folks using that flag would need a >> Python-level loop, but folks using the kernel's CSPRNG wouldn't. > > Do you mean return the 4000 bytes and *ignores* the error? I'm not clear on the ways that GRNG_RANDOM can fail (other than EINTR), so I'm not sure. I just think we have a fair bit of leeway for the behaviour of that flag to be less user friendly than the default behaviour, since the "may block unexpectedly during normal execution" behaviour is already inherently user unfriendly :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Sat Jul 30 02:19:24 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 30 Jul 2016 08:19:24 +0200 Subject: [Security-sig] PEP 524: Make os.urandom() blocking on Linux (version 3) In-Reply-To: References: <20160729093155.2fec8274.barry@wooz.org> Message-ID: Le 30 juil. 2016 6:55 AM, "Nick Coghlan" a ?crit : > I'm not clear on the ways that GRNG_RANDOM can fail (other than > EINTR) You just gave an example no? :-) Signals are common on UNIX, not only to quit an application. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: