[Security-sig] PEP 522: Allow BlockingIOError in security sensitive APIs on Linux

[Victor Stinner]

> The new exception would potentially be encountered in the following situations:
>
> * Python code calling these APIs during Linux system initialization

I'm not sure that there is such use case in practice.

Can you please try to describe an use case where you would need
blocking system urandom *during the Python initialization*?

It looks like my use case 1, but I consider that os.urandom() is *not*
called on such use case:
https://haypo-notes.readthedocs.io/pep_random.html#use-case-1-init-script


> * Python code running on improperly initialized Linux systems (e.g. embedded
>   hardware without adequate sources of entropy to seed the system random number
>   generator, or Linux VMs that aren't configured to accept entropy from the
>   VM host)

If the program doesn't use os.urandom(), well, we don't care, there is
no issue :-)

IMO the interesting use case is when the application really requires
secure secret. That's my use case 2, a web server:
https://haypo-notes.readthedocs.io/pep_random.html#use-case-2-web-server

I chose to not give the choice to the developer and block on such
case. IMO it's accepable because the application should not have to
wait forever for urandom.

> Changing ``os.urandom()`` on Linux
> ----------------------------------
>
> This PEP proposes that in Python 3.6+, ``os.urandom()`` be updated to call
> the new Linux ``getrandom()`` syscall in non-blocking mode if available and
> raise ``BlockingIOError: system random number generator is not ready`` if
> the kernel reports that the call would block.

To be clear, the behaviour is unchanged on other platforms, right?

I'm just trying to understand the scope of the PEP. It looks like as
mine, it is written for Linux. (Even if other platforms may implement
the same behaviour later, if needed.)

If it's deliberate to restrict to Linux, you may be more explicit at
least in the abstract.

--

By the way, are you aware of other programming languages or
applications using an exception when random would block? (It's not a
requirement, I'm just curious.)




> By contrast, if ``BlockingIOError`` is raised in those situations, then
> developers using Python 3.6+ can easily choose their desired behaviour:
>
> 1. Loop until the call succeeds (security sensitive)

Is this case different from a blocking os.urandom()?


> 2. Switch to using the random module (non-security sensitive)

Hum, I disagree on this point. I don't think that you should start
with os.urandom() to fallback on random.

In fact, I only know *one* use case for this: create the random.Random
instance when the random module is imported.

In my PEP, I proposed to have a special case for random.Random
constructor, implemented in C (to not have to expose anything at the
Python level).


> 3. Switch to reading ``/dev/urandom`` directly (non-security sensitive)

It is what I propose for the random.Random constructor when the random
module is imported.

Again, the question is if there is a real use case for it. And if yes,
if the use case common enough to justify the change?


The extreme case is that all applications using os.urandom() would
need to be modifiy to add a try/except BlockingIOError. I only
exagerate to try to understand the impact of your PEP. I only that
only a few applications will use such try/except in practice.

As I tried to explain in my PEP, with Python 3.5.2, "the bug" (block
on random) became very unlikely.


> Issuing a warning for potentially predictable internal hash initialization

I don't recall Python logging warnings for similar issues. But I don't
recall similar issues neither :-)


> The challenge for internal hash initialization is that it might be very
> important to initialize SipHash with a reliably unpredictable random seed
> (for processes that are exposed to potentially hostile input) or it might be
> totally unimportant (for processes that never have to deal with untrusted data).

>From what I read, /dev/urandom is good even before it is considered as
initialized, because the kernel collects various data, but don't
increase the entropy estimator.

I'm not completely convinced that a warning is needed. I'm not against
it neither. I am doubtful. :-)

Well, let's say that we have a warning. What should the user do in
such case? Is it an advice to dig the urandom issue and try to get
more entropy?

The warning is for users, no? I imagine that an application can work
perfectly for the developer, but only emit the warning for some users
depending how the deploy their application.


> However, at the same time, since Python has no way to know whether any given
> invocation needs to handle untrusted data, when the default SipHash
> initialization fails this *might* indicate a genuine security problem, which
> should not be allowed to pass silently.

An alternative would be to provide a read-only flag which would
indicate if the hash secret is considered as "secure" or not.

Applications considered by security would check the flag and decide
themself to emit a warning or not.


> Accordingly, if internal hash initialization needs to fall back to a potentially
> predictable seed due to the system random number generator not being ready, it
> will also emit a warning message on ``stderr`` to say that the system random
> number generator is not available and that processing potentially hostile
> untrusted data should be avoided.

I know that many of you disagree with me, but I'm not sure that the
hash DoS is an important issue.

We should not overestimate the importance of this vulnerability.


> Affected security sensitive applications
> ----------------------------------------
>
> Security sensitive applications would need to either change their system
> configuration so the application is only started after the operating system
> random number generator is ready for security sensitive operations, or else
> change their code to busy loop until the operating system is ready::
>
>     def blocking_urandom(num_bytes):
>         while True:
>             try:
>                 return os.urandom(num_bytes)
>             except BlockingIOError:
>                 pass

Such busy-loop may use a lot of CPU :-/ You need a time.sleep() or
something like that, no?

A blocking os.urandom() doesn't have such issue ;-)

Is it possible that os.urandom() works, but the following os.urandom()
call raises a BlockingIOError? If yes, there is an issue with "partial
read", we should uses a dedicated exception to return partial data.

Hopefully, I understood that the issue doesn't occur in pratice.
os.urandom() starts with BlockingIOError. But once it "works", it will
work forever. Well, at least on Linux.

I don't know how Solaris behaves. I hope that it behaves as Linux
(once it works, it always works). At least, I see that Solaris
getrandom() can also fails with EAGAIN.


> Affected non-security sensitive applications
> --------------------------------------------
>
> Non-security sensitive applications that don't want to assume access to
> ``/dev/urandom`` (or assume a non-blocking implementation of that device)
> can be updated to use the ``random`` module as a fallback option::
>
>     def pseudorandom_fallback(num_bytes):
>         try:
>             return os.urandom(num_bytes)
>         except BlockingIOError:
>             random.getrandbits(num_bytes*8).to_bytes(num_bytes, "little")
>
> Depending on the application, it may also be appropriate to skip accessing
> ``os.urandom`` at all, and instead rely solely on the ``random`` module.

Hum, I dislike such change. It overcomplicates applications for a corner-case.

If you use os.urandom(), you already expect security. I prefer to
simplify use cases to two cases: (1) you really need security (2) you
really don't care of security. If you don't care, use directly the
random module. Don't bother with os.urandom() nor having to add
try/except BlockingIOError. No?

I *hope* that a regular application will never see BlockingIOError on
os.urandom() in the wild.


> Affected Linux specific non-security sensitive applications
> -----------------------------------------------------------
>
> Non-security sensitive applications that don't need to worry about cross
> platform compatibility and are willing to assume that ``/dev/urandom`` on
> Linux will always retain its current behaviour can be updated to access
> ``/dev/urandom`` directly::
>
>     def dev_urandom(num_bytes):
>         with open("/dev/urandom", "rb") as f:
>             return f.read(num_bytes)

Again, I'm against adding such complexity for a corner case. Just use
os.urandom().



> For additional background details beyond those captured in this PEP, also see
> Victor Stinner's summary at http://haypo-notes.readthedocs.io/pep_random.html

Oh, I didn't expect to have references to my document :-) I moved it to:
https://haypo-notes.readthedocs.io/summary_python_random_issue.html

http://haypo-notes.readthedocs.io/pep_random.html is now really a PEP ;-)

Victor