[Python-Dev] BDFL ruling request: should we block forever waiting for high-quality random bits?

[Barry Warsaw]

On Jun 15, 2016, at 11:52 PM, Larry Hastings wrote:

>Well, 3.5.2 hasn't happened yet.  So if you see it as still being broken,
>please speak up now.

In discussion with other Ubuntu developers, several salient points were raised.

The documentation for os.urandom() in 3.5.2rc1 doesn't make sense:

    On Linux, getrandom() syscall is used if available and the urandom entropy
    pool is initialized (getrandom() does not block). On a Unix-like system
    this will query /dev/urandom.

Perhaps better would be:

    Where available, the getrandom() syscall is used (with the GRND_NONBLOCK
    flag) if available and the urandom entropy pool is initialized.  When
    getrandom() returns EAGAIN because of insufficient entropy, fallback to
    reading from /dev/urandom.  When the getrandom() syscall is unavailable on
    other Unix-like systems, this will query /dev/urandom.

It's actually a rather twisty maze of code to verify these claims, and I'm
nearly certain we don't have any tests to guarantee this is what actually
happens in those cases, so there are many caveats.

This means that an experienced developer can no longer just `man urandom` to
understand the unique operational behavior of os.urandom() on their platform,
but instead would be forced to actually read our code to find out what's
actually happening when/if things break.

It is unacceptable if any new exceptions are raised when insufficient entropy
is available.  Python 3.4 essentially promises that "if only crap entropy is
available, you'll get crap, but at least it won't block and no exceptions are
raised".  Proper backward compatibility requires the same in 3.5 and beyond.
Are we sure that's still the case?

Using the system call *may* be faster in the we-have-good-entropy-case, but it
will definitely be slower in the we-don't-have-good-entropy-case (because of
the fallback logic).  Maybe that doesn't matter in practice but it's worth
noting.

>Why do you call it only "semi-fixed"?  As far as I understand it, the
>semantics of os.urandom() in 3.5.2rc1 are indistinguishable from reading from
>/dev/urandom directly, except it may not need to use a file handle.

Semi-fixed because os.urandom() will still not be strictly backward compatible
between Python 3.5.2 and 3.4.

*If* we can guarantee that os.urandom() will never block or raise an exception
when only poor entropy is available, then it may be indeed indistinguishably
backward compatible for most if not all cases.

Cheers,
-Barry