[Patches] [ python-Patches-934711 ] platform-specific entropy

Sun Apr 25 20:49:08 EDT 2004

Patches item #934711, was opened at 2004-04-14 00:05
Message generated for change (Comment added) made by nickm
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=934711&group_id=5470

Category: Core (C code)
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: Trevor Perrin (trevp)
Assigned to: Nobody/Anonymous (nobody)
Summary: platform-specific entropy

Initial Comment:

This is a very simple module for accessing platform-
specific entropy sources.  

On Win32, it uses CryptGenRandom.  Otherwise, it tries 
to use /dev/urandom.  If it can't find that, it raises 
NotImplementedError.

>>> import entropy
>>> entropy.entropy(10)
'\x99/\xbd\x8e\xb0\xfbz/\xf6\xe9'

To compile for Win32, you have to link with "advapi32".  
The /dev/urandom code has not been tested.

Issues:

 - I believe this works with all versions of Windows later 
than Win95.  But I'm not sure.

 - there's overhead in opening/closing the source.  On 
Win32 at least, opening it is slower than reading from it -
 it seems to take around 1/5th of a millisecond (i.e. I 
can make 5000 calls per second).  I think this is okay; 
you can make fewer calls and buffer the results, or seed 
your own PRNG.  However, we could make it more 
object-based, where you open an entropy-source, and 
then repeatedly read from it.

 - It would be nice if there was some attribute that told 
you what source you were using, so you could display 
it, and in case you trust /dev/urandom but not Win32's 
CryptoAPI, for example.

----------------------------------------------------------------------

Comment By: Nick Mathewson (nickm)
Date: 2004-04-25 20:49

Message:
Logged In: YES 
user_id=499

Thanks for the reply!

- As for /dev/*random -- yes, I believe you are right, and
/dev/urandom is almost always what you want.  I haven't been
able to find a platform that has one of the others, but
lacks /dev/urandom.

- I can't find a statement on the page you link about using
CRYPT_VERIFYCONTEXT that way, but you may well be right anway.

- One more important issue: It is a bad idea to use stdio
(C's 'fopen', Python's builtin 'open') to read from
/dev/urandom.  Most stdio implementation buffer data; on my
GNU/Linux box, when I call open('/dev/urandom').read(10), my
underlying fread() function sucks 4096 bytes into memory. 
(It does other weird stuff too, including calls to stat64,
mmap, and others.)  This has proved to be a problem in the
past, especially when running on systems with heavy user
process limits.  Instead, it is a better idea to use the
open syscall directly (open in C, os.open in Python).

----------------------------------------------------------------------

Comment By: Trevor Perrin (trevp)
Date: 2004-04-25 20:08

Message:
Logged In: YES 
user_id=973611

Thanks for the comments! - 

> - According to MSDN, CryptGenRandom exists on Win98 and
> later, and on Win95 OSR2, and on Win95 with IE 3.something
> or later.

I'm uploading a new version that fails gracefully on old
Win95s (that's the only change).

> - It's necessary on some platforms, and for some
> applications, to use a device other than /dev/urandom. 
> (Some high-security code demands /dev/random; some 
> OpenBSD people swear by /dev/arandom; and so on.)

My understanding is that /dev/random should only be used in
exceptionally rare cases, if at all.  You can turn up
several posts by David Wagner, a respected cryptographer,
about this.  For example:
http://tinyurl.com/2z2fx

In any case, if you really want /dev/random, or one of the
OpenBSD variants (arandom, srandom, prandom, etc.), it's
easy to do it yourself: open("/dev/random").read().

So I think we should ignore these and stick with
/dev/urandom, since it's the easiest-to-use (non-blocking)
and most portable (unless there are systems that don't offer
it?).

> - Maybe it would be a good idea to only implement the
> windows CryptGenRandom part in C, and implement the Unix
> part in Python.

That's not a bad idea - I sorta think this function should
be placed in the 'os' module, instead of its own module.  So
we could put the /dev/urandom code in 'os.py', and allow
more specific code in, e.g., posixmodule.c to override it.  

We could also add a variable 'os.entropySource' which would
return '/dev/urandom', or 'CryptoAPI', or whatever.

> - According to the MSDN documentation for
> CryptAcquireContext, if your first call fails, you're
> supposed to retry with a final argument of 
> CRYPT_NEWKEYSET before you report an error.

I'm pretty sure using CRYPT_VERIFYCONTEXT eliminates the
need for that:
http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q238187&ID=KB;EN-US;Q238187

----------------------------------------------------------------------

Comment By: Nick Mathewson (nickm)
Date: 2004-04-25 14:55

Message:
Logged In: YES 
user_id=499

This patch would be tremendously valuable to me.  I've had
to maintain similar code for a couple of projects, and
having this functionality in Python would make my life one
step similar.

A few comments:
- According to MSDN, CryptGenRandom exists on Win98 and
later, and on Win95 OSR2, and on Win95 with IE 3.something
or later.

- It's necessary on some platforms, and for some
applications, to use a device other than /dev/urandom. 
(Some high-security code demands /dev/random; some OpenBSD
people swear by /dev/arandom; and so on.)

- Maybe it would be a good idea to only implement the
windows CryptGenRandom part in C, and implement the Unix
part in Python.  Then you could expose the windows code as
(say) "entopy.win_entropy(nBytes)", the unix part as (say)
"entropy.unix_entropy(nBytes, file='/dev/urandom')", and
have "entropy.entropy(nBytes)" be a cross-platform wrapper.
 This would cut down on the amount of C you need to add;
make it easier to switch entropy devices; provide better
errors when /dev/urandom is unreadable; and provide users
the option to trust only certain entropy sources.

- I believe that you're right not to worry too much about
the performance here.

- According to the MSDN documentation for
CryptAcquireContext, if your first call fails, you're
supposed to retry with a final argument of CRYPT_NEWKEYSET
before you report an error.  I don't know when/if this is
necessary, or whether there are hidden gotchas.

Once again, this patch is a great idea, and I heartily hope
that it gets in!

----------------------------------------------------------------------

Comment By: Trevor Perrin (trevp)
Date: 2004-04-14 02:19

Message:
Logged In: YES 
user_id=973611

Just a thought - this might make sense as a function within 
the 'os' module, instead of its own module.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=934711&group_id=5470