From alex.gaynor at gmail.com Wed Jan 8 01:57:21 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 7 Jan 2014 16:57:21 -0800 Subject: [Cryptography-dev] First release! Message-ID: Hey all, At this point we have one item on the "First release milestone". If you think anything is missing please add it to the milestone (or if you don't have permissions to do that, mention it here). Once we're done with all of those we should coordinate doing that release. I think we want our first release to be pretty low key -- enough to let JP use it for pyOpenSSL, and get feedback on the API. And we can do a big push to get adoption with our second release. Thoughts? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rgbkrk at gmail.com Wed Jan 8 04:45:10 2014 From: rgbkrk at gmail.com (Kyle Kelley) Date: Tue, 7 Jan 2014 21:45:10 -0600 Subject: [Cryptography-dev] First release! In-Reply-To: References: Message-ID: Yay! I've been watching this mailing list in hopes that something I'm working on can use what you're all building here (instead of pycrypto, etc.). I'm especially looking forward to public/private key conversion and any/all utilities for public key algorithms. -- Kyle On Tue, Jan 7, 2014 at 6:57 PM, Alex Gaynor wrote: > Hey all, > > At this point we have one item on the "First release milestone". If you > think anything is missing please add it to the milestone (or if you don't > have permissions to do that, mention it here). > > Once we're done with all of those we should coordinate doing that release. > I think we want our first release to be pretty low key -- enough to let JP > use it for pyOpenSSL, and get feedback on the API. And we can do a big push > to get adoption with our second release. > > Thoughts? > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From randall.leeds at gmail.com Thu Jan 9 02:03:59 2014 From: randall.leeds at gmail.com (Randall Leeds) Date: Wed, 8 Jan 2014 17:03:59 -0800 Subject: [Cryptography-dev] Introductions and contributions Message-ID: Hello, list! Just saw Alex mention the first release on Twitter -- nice work! This is the first I've heard of this lib. I've got a decent grasp of DSA (or had, once upon a time I proved all the relevant theorems as an exercise), so I'm thinking of plugging myself in and attempting #230 [1]. If you've got recommendations for other areas that are more pressing, though, I'm happy to assist however I can. -Randall [1] https://github.com/pyca/cryptography/issues/230 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dreid at dreid.org Thu Jan 9 02:19:58 2014 From: dreid at dreid.org (David Reid) Date: Wed, 8 Jan 2014 17:19:58 -0800 Subject: [Cryptography-dev] Introductions and contributions In-Reply-To: References: Message-ID: Welcome, The big open questions around DSA (and RSA for that matter) is all about the API, here is some of the relevant discussion: https://github.com/pyca/cryptography/pull/292 & https://github.com/pyca/cryptography/pull/290 That expresses the somewhat vague goal of being able to generate an RSA key with OpenSSL and lets you use it with CommonCrypto. It's unclear if that is actually a useful property of the API however. It also isn't entirely clear what if any the common API between using RSA and DSA is. So yes, please, any input on these questions would be great. -David On Wed, Jan 8, 2014 at 5:03 PM, Randall Leeds wrote: > Hello, list! > > Just saw Alex mention the first release on Twitter -- nice work! This is > the first I've heard of this lib. > > I've got a decent grasp of DSA (or had, once upon a time I proved all the > relevant theorems as an exercise), so I'm thinking of plugging myself in > and attempting #230 [1]. > > If you've got recommendations for other areas that are more pressing, > though, I'm happy to assist however I can. > > -Randall > > [1] https://github.com/pyca/cryptography/issues/230 > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hs at ox.cx Fri Jan 17 21:59:04 2014 From: hs at ox.cx (Hynek Schlawack) Date: Fri, 17 Jan 2014 21:59:04 +0100 Subject: [Cryptography-dev] 0.2 Message-ID: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> Dear fellow makers of poorest life choices, I?d like to speak about our next release. Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 Thoughts? ?h -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 841 bytes Desc: OpenPGP digital signature URL: From alex.gaynor at gmail.com Fri Jan 17 22:00:38 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 17 Jan 2014 15:00:38 -0600 Subject: [Cryptography-dev] 0.2 In-Reply-To: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> Message-ID: +1 from me. My original goals were asymmetric and common crypto for 0.2, but I'm happy to release early and release early if it'll help improve the security of the internet ;) (melodrama ftw!) Alex On Fri, Jan 17, 2014 at 2:59 PM, Hynek Schlawack wrote: > Dear fellow makers of poorest life choices, > > I?d like to speak about our next release. > > Concretely I?d like to get out 0.2 as soon as JP gives us feedback on > https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be > considered finished. I believe ECDH will add enough value on it?s own to > warrant a release; primarily it will also help move PyOpenSSL and maybe > even Twisted forward. Having that done before PyCon would be a *great* > achievement. > > As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = > https://gist.github.com/hynek/2f0acd65a34523028168 > > Thoughts? > ?h > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.l.kehrer at gmail.com Fri Jan 17 22:06:54 2014 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Fri, 17 Jan 2014 15:06:54 -0600 Subject: [Cryptography-dev] 0.2 In-Reply-To: References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> Message-ID: <1A7AB688A9DB481D950E0F21485DC424@gmail.com> I?m also happy to do a release that encompasses this functionality while we work on our other items (which we can roll over to a third release milestone). On Friday, January 17, 2014 at 3:00 PM, Alex Gaynor wrote: > +1 from me. My original goals were asymmetric and common crypto for 0.2, but I'm happy to release early and release early if it'll help improve the security of the internet ;) (melodrama ftw!) > > Alex > > > On Fri, Jan 17, 2014 at 2:59 PM, Hynek Schlawack wrote: > > Dear fellow makers of poorest life choices, > > > > I?d like to speak about our next release. > > > > Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. > > > > As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 > > > > Thoughts? > > ?h > > _______________________________________________ > > Cryptography-dev mailing list > > Cryptography-dev at python.org (mailto:Cryptography-dev at python.org) > > https://mail.python.org/mailman/listinfo/cryptography-dev > > > > > > -- > "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org (mailto:Cryptography-dev at python.org) > https://mail.python.org/mailman/listinfo/cryptography-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Jan 17 22:10:39 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 17 Jan 2014 16:10:39 -0500 Subject: [Cryptography-dev] 0.2 In-Reply-To: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> Message-ID: No! Ok fine ;) > On Jan 17, 2014, at 3:59 PM, "Hynek Schlawack" wrote: > > Dear fellow makers of poorest life choices, > > I?d like to speak about our next release. > > Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. > > As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 > > Thoughts? > ?h > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev From jean-paul at hybridcluster.com Fri Jan 17 23:08:23 2014 From: jean-paul at hybridcluster.com (Jean-Paul Calderone) Date: Fri, 17 Jan 2014 17:08:23 -0500 Subject: [Cryptography-dev] 0.2 In-Reply-To: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> Message-ID: <52D9A9D7.9030801@hybridcluster.com> On 01/17/2014 03:59 PM, Hynek Schlawack wrote: > Dear fellow makers of poorest life choices, > > I?d like to speak about our next release. > > Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. > > As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 > > Thoughts? > ?h > FWIW, I'm completely unprepared to provide feedback related to new pyOpenSSL features. I have known bugs in 0.14a2 to fix and, apart from the people on this list who I bugged on IRC to test out 0.14a2, no one has provided any feedback on the alpha so far. If that continues for a couple weeks then I'll just have to go ahead with the release anyway, but given that the entire implementation has been rewritten for this release I was really hoping to get at least a little bit of pre-release community feedback. For planning purposes, I suggest you expect pyOpenSSL 0.14 final no earlier than Feb 1 and perhaps as late as the middle or end of Feb. Sorry and thanks. Jean-Paul -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From alexs at prol.etari.at Fri Jan 17 23:17:44 2014 From: alexs at prol.etari.at (Alex Stapleton) Date: Fri, 17 Jan 2014 22:17:44 +0000 Subject: [Cryptography-dev] 0.2 In-Reply-To: <143a2465960.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> Message-ID: <143a247e7d0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> By "our ECDSA API" do you mean cryptography or PyOpenSSL? Bindings for PyOpenSSL are relatively easy so that sounds entirely sensible :-) (Apologies if this next part turns out to be OT.) On the other hand the things blocking asymmetric progress in cryptography are as I understand it: 1. Proposals and review of signing interfaces. 2. Thread safety of key generation due to the RNG. (Maybe other bits effected by this too.) 3. Key serialisation. (PKCS needs PBES/PBKDF really, and some changes to our symmetric interface maybe.) 4. Refactoring of OpenSSL error handling. 5. Fork safety mitigation for the OpenSSL RNG. ECDSA might be OK without 3, which also makes 4 a lot easier as it avoids complex error paths in OpenSSL. I think the osrandom RNG patch for 5 might turn out to fix 2 for now as well? Some of these issues probably effect usage in PyOpenSSL and twisted too so maybe we can borrow some solutions from there :-) Sent with AquaMail for Android http://www.aqua-mail.com On 17 January 2014 20:59:04 "Hynek Schlawack" wrote: > Dear fellow makers of poorest life choices, > > I?d like to speak about our next release. > > Concretely I?d like to get out 0.2 as soon as JP gives us feedback on > https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered > finished. I believe ECDH will add enough value on it?s own to warrant a > release; primarily it will also help move PyOpenSSL and maybe even Twisted > forward. Having that done before PyCon would be a *great* achievement. > > As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = > https://gist.github.com/hynek/2f0acd65a34523028168 > > Thoughts? > ?h From hs at ox.cx Sat Jan 18 09:14:31 2014 From: hs at ox.cx (Hynek Schlawack) Date: Sat, 18 Jan 2014 09:14:31 +0100 Subject: [Cryptography-dev] 0.2 In-Reply-To: <143a247e7d0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <143a247e7d0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: On 17 Jan 2014, at 23:17, Alex Stapleton wrote: > By "our ECDSA API" do you mean cryptography or PyOpenSSL? EC*DH* > Bindings for PyOpenSSL are relatively easy so that sounds entirely > sensible :-) > > (Apologies if this next part turns out to be OT.) > > On the other hand the things blocking asymmetric progress in > cryptography are as I understand it: > > 1. Proposals and review of signing interfaces. > 2. Thread safety of key generation due to the RNG. (Maybe other bits > effected by this too.) > 3. Key serialisation. (PKCS needs PBES/PBKDF really, and some changes > to our symmetric interface maybe.) > 4. Refactoring of OpenSSL error handling. > 5. Fork safety mitigation for the OpenSSL RNG. > ECDSA might be OK without 3, which also makes 4 a lot easier as it > avoids complex error paths in OpenSSL. I think the osrandom RNG patch > for 5 might turn out to fix 2 for now as well? > > Some of these issues probably effect usage in PyOpenSSL and twisted > too so maybe we can borrow some solutions from there :-) > > > > Sent with AquaMail for Android > http://www.aqua-mail.com > > > On 17 January 2014 20:59:04 "Hynek Schlawack" wrote: >> Dear fellow makers of poorest life choices, >> >> I?d like to speak about our next release. >> >> Concretely I?d like to get out 0.2 as soon as JP gives us feedback >> on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be >> considered finished. I believe ECDH will add enough value on it?s >> own to warrant a release; primarily it will also help move PyOpenSSL >> and maybe even Twisted forward. Having that done before PyCon would >> be a *great* achievement. >> >> As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = >> https://gist.github.com/hynek/2f0acd65a34523028168 >> >> Thoughts? >> ?h > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev From alexs at prol.etari.at Sat Jan 18 09:58:54 2014 From: alexs at prol.etari.at (Alex Stapleton) Date: Sat, 18 Jan 2014 08:58:54 +0000 Subject: [Cryptography-dev] 0.2 In-Reply-To: References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <143a247e7d0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: <143a492ed28.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Heh I fail. Too many damn abbreviations :-) I think that a 0.2 aimed at PyOpenSSL is a good idea. Are there any other issues need sorting out? I can't remember what state the locking stuff ended up in. Sent with AquaMail for Android http://www.aqua-mail.com On 18 January 2014 08:14:31 "Hynek Schlawack" wrote: > On 17 Jan 2014, at 23:17, Alex Stapleton wrote: > > > By "our ECDSA API" do you mean cryptography or PyOpenSSL? > > EC*DH* > > > > Bindings for PyOpenSSL are relatively easy so that sounds entirely > sensible :-) > > > > (Apologies if this next part turns out to be OT.) > > > > On the other hand the things blocking asymmetric progress in cryptography > are as I understand it: > > > > 1. Proposals and review of signing interfaces. > > 2. Thread safety of key generation due to the RNG. (Maybe other bits > effected by this too.) > > 3. Key serialisation. (PKCS needs PBES/PBKDF really, and some changes to > our symmetric interface maybe.) > > 4. Refactoring of OpenSSL error handling. > > 5. Fork safety mitigation for the OpenSSL RNG. > > ECDSA might be OK without 3, which also makes 4 a lot easier as it avoids > complex error paths in OpenSSL. I think the osrandom RNG patch for 5 might > turn out to fix 2 for now as well? > > > > Some of these issues probably effect usage in PyOpenSSL and twisted too > so maybe we can borrow some solutions from there :-) > > > > > > > > Sent with AquaMail for Android > > http://www.aqua-mail.com > > > > > > On 17 January 2014 20:59:04 "Hynek Schlawack" wrote: > >> Dear fellow makers of poorest life choices, > >> > >> I?d like to speak about our next release. > >> > >> Concretely I?d like to get out 0.2 as soon as JP gives us feedback on > https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered > finished. I believe ECDH will add enough value on it?s own to warrant a > release; primarily it will also help move PyOpenSSL and maybe even Twisted > forward. Having that done before PyCon would be a *great* achievement. > >> > >> As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = > https://gist.github.com/hynek/2f0acd65a34523028168 > >> > >> Thoughts? > >> ?h > > > > > > _______________________________________________ > > Cryptography-dev mailing list > > Cryptography-dev at python.org > > https://mail.python.org/mailman/listinfo/cryptography-dev > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev From hs at ox.cx Sat Jan 18 11:07:37 2014 From: hs at ox.cx (Hynek Schlawack) Date: Sat, 18 Jan 2014 11:07:37 +0100 Subject: [Cryptography-dev] 0.2 In-Reply-To: <143a492ed28.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <143a247e7d0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> <143a492ed28.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: <4BEDB169-C74F-4176-B647-A354DB3FEC56@ox.cx> On 18 Jan 2014, at 9:58, Alex Stapleton wrote: > Heh I fail. Too many damn abbreviations :-) > > I think that a 0.2 aimed at PyOpenSSL is a good idea. Are there any > other issues need sorting out? I can't remember what state the locking > stuff ended up in. As JP wrote, the main issue is feedback to his alpha (maybe that should be raised on the Twisted list too). Other than that, there are some OS X issues: - the two related to using an OpenSSL that Python is *not* linked against you bravely suffered through with me (https://github.com/pyca/pyopenssl/issues/7 + https://github.com/pyca/pyopenssl/issues/3) :) - the ReadWrite test fail ( https://github.com/pyca/pyopenssl/issues/8 ). It would be great to have OS X Travis for these; I seem to recall that Donald tried to ping them but they were offline or something? Cheers, ?h From jean-paul at hybridcluster.com Sat Jan 18 17:09:11 2014 From: jean-paul at hybridcluster.com (Jean-Paul Calderone) Date: Sat, 18 Jan 2014 11:09:11 -0500 Subject: [Cryptography-dev] 0.2 In-Reply-To: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> Message-ID: <52DAA727.70704@hybridcluster.com> On 01/17/2014 03:59 PM, Hynek Schlawack wrote: > Dear fellow makers of poorest life choices, > > I?d like to speak about our next release. > > Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. > > As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 I've just been reminded that some of the required initialization APIs are not in 0.1 so pyOpenSSL 0.14 will need cryptography 0.2. Jean-Paul -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From glyph at twistedmatrix.com Sun Jan 19 00:56:48 2014 From: glyph at twistedmatrix.com (GMail) Date: Sat, 18 Jan 2014 15:56:48 -0800 Subject: [Cryptography-dev] 0.2 In-Reply-To: <52DAA727.70704@hybridcluster.com> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <52DAA727.70704@hybridcluster.com> Message-ID: <0803E63D-192F-4AD5-A394-4EF5FD9CA0C1@twistedmatrix.com> On Jan 18, 2014, at 8:09 AM, Jean-Paul Calderone wrote: > On 01/17/2014 03:59 PM, Hynek Schlawack wrote: >> Dear fellow makers of poorest life choices, >> >> I?d like to speak about our next release. >> >> Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. >> >> As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 > > I've just been reminded that some of the required initialization APIs > are not in 0.1 so pyOpenSSL 0.14 will need cryptography 0.2. Which specific APIs are those? -glyph From jean-paul at hybridcluster.com Sun Jan 19 03:32:35 2014 From: jean-paul at hybridcluster.com (Jean-Paul Calderone) Date: Sat, 18 Jan 2014 21:32:35 -0500 Subject: [Cryptography-dev] 0.2 In-Reply-To: <0803E63D-192F-4AD5-A394-4EF5FD9CA0C1@twistedmatrix.com> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <52DAA727.70704@hybridcluster.com> <0803E63D-192F-4AD5-A394-4EF5FD9CA0C1@twistedmatrix.com> Message-ID: <52DB3943.4060806@hybridcluster.com> On 01/18/2014 06:56 PM, GMail wrote: > On Jan 18, 2014, at 8:09 AM, Jean-Paul Calderone wrote: > >> On 01/17/2014 03:59 PM, Hynek Schlawack wrote: >>> Dear fellow makers of poorest life choices, >>> >>> I?d like to speak about our next release. >>> >>> Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. >>> >>> As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 >> I've just been reminded that some of the required initialization APIs >> are not in 0.1 so pyOpenSSL 0.14 will need cryptography 0.2. > Which specific APIs are those? They *were* ERR_load_crypto_strings() and ERR_load_SSL_strings but reaperhulk pointed out that cryptography 0.1 has SSL_load_error_strings so I've adjusted pyOpenSSL to use that instead. So now it again appears as though pyOpenSSL 0.14 will be satisfied with cryptography 0.1. Jean-Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From glyph at twistedmatrix.com Sun Jan 19 03:35:27 2014 From: glyph at twistedmatrix.com (Glyph) Date: Sat, 18 Jan 2014 18:35:27 -0800 Subject: [Cryptography-dev] 0.2 In-Reply-To: <52DB3943.4060806@hybridcluster.com> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <52DAA727.70704@hybridcluster.com> <0803E63D-192F-4AD5-A394-4EF5FD9CA0C1@twistedmatrix.com> <52DB3943.4060806@hybridcluster.com> Message-ID: <95EC525C-3DC8-4CB4-A774-225F133EC42F@twistedmatrix.com> On Jan 18, 2014, at 6:32 PM, Jean-Paul Calderone wrote: > On 01/18/2014 06:56 PM, GMail wrote: >> >> On Jan 18, 2014, at 8:09 AM, Jean-Paul Calderone wrote: >> >>> On 01/17/2014 03:59 PM, Hynek Schlawack wrote: >>>> Dear fellow makers of poorest life choices, >>>> >>>> I?d like to speak about our next release. >>>> >>>> Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. >>>> >>>> As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 >>> I've just been reminded that some of the required initialization APIs >>> are not in 0.1 so pyOpenSSL 0.14 will need cryptography 0.2. >> Which specific APIs are those? > > They were ERR_load_crypto_strings() and ERR_load_SSL_strings but reaperhulk pointed out that cryptography 0.1 has SSL_load_error_strings so I've adjusted pyOpenSSL to use that instead. So now it again appears as though pyOpenSSL 0.14 will be satisfied with cryptography 0.1. Thanks. I was asking mostly for my own edification, but good to know they were in fact wrapped :-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Sun Jan 19 03:36:13 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 18 Jan 2014 20:36:13 -0600 Subject: [Cryptography-dev] 0.2 In-Reply-To: <95EC525C-3DC8-4CB4-A774-225F133EC42F@twistedmatrix.com> References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <52DAA727.70704@hybridcluster.com> <0803E63D-192F-4AD5-A394-4EF5FD9CA0C1@twistedmatrix.com> <52DB3943.4060806@hybridcluster.com> <95EC525C-3DC8-4CB4-A774-225F133EC42F@twistedmatrix.com> Message-ID: JP -- that means it's not your intent to get the ECDHE stuff in for 0.14 then? (That's fine, just want to be explicit) Alex On Sat, Jan 18, 2014 at 8:35 PM, Glyph wrote: > > On Jan 18, 2014, at 6:32 PM, Jean-Paul Calderone < > jean-paul at hybridcluster.com> wrote: > > On 01/18/2014 06:56 PM, GMail wrote: > > On Jan 18, 2014, at 8:09 AM, Jean-Paul Calderone wrote: > > > On 01/17/2014 03:59 PM, Hynek Schlawack wrote: > > Dear fellow makers of poorest life choices, > > I?d like to speak about our next release. > > Concretely I?d like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it?s own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. > > As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 > > I've just been reminded that some of the required initialization APIs > are not in 0.1 so pyOpenSSL 0.14 will need cryptography 0.2. > > Which specific APIs are those? > > > They *were* ERR_load_crypto_strings() and ERR_load_SSL_strings but > reaperhulk pointed out that cryptography 0.1 has SSL_load_error_stringsso I've adjusted pyOpenSSL to use that instead. So now it again appears as > though pyOpenSSL 0.14 will be satisfied with cryptography 0.1. > > > Thanks. I was asking mostly for my own edification, but good to know they > were in fact wrapped :-). > > -glyph > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-paul at hybridcluster.com Sun Jan 19 14:00:11 2014 From: jean-paul at hybridcluster.com (Jean-Paul Calderone) Date: Sun, 19 Jan 2014 08:00:11 -0500 Subject: [Cryptography-dev] 0.2 In-Reply-To: References: <218A54EE-724D-4F23-8463-2C60D53ED4BD@ox.cx> <52DAA727.70704@hybridcluster.com> <0803E63D-192F-4AD5-A394-4EF5FD9CA0C1@twistedmatrix.com> <52DB3943.4060806@hybridcluster.com> <95EC525C-3DC8-4CB4-A774-225F133EC42F@twistedmatrix.com> Message-ID: <52DBCC5B.8040505@hybridcluster.com> On 01/18/2014 09:36 PM, Alex Gaynor wrote: > JP -- that means it's not your intent to get the ECDHE stuff in for > 0.14 then? (That's fine, just want to be explicit) > > Alex Yep. Jean-Paul > > > On Sat, Jan 18, 2014 at 8:35 PM, Glyph > wrote: > > > On Jan 18, 2014, at 6:32 PM, Jean-Paul Calderone > > > wrote: > >> On 01/18/2014 06:56 PM, GMail wrote: >>> On Jan 18, 2014, at 8:09 AM, Jean-Paul Calderone wrote: >>> >>>> On 01/17/2014 03:59 PM, Hynek Schlawack wrote: >>>>> Dear fellow makers of poorest life choices, >>>>> >>>>> I'd like to speak about our next release. >>>>> >>>>> Concretely I'd like to get out 0.2 as soon as JP gives us feedback on https://github.com/pyca/pyopenssl/pull/9 and our ECDH API can be considered finished. I believe ECDH will add enough value on it's own to warrant a release; primarily it will also help move PyOpenSSL and maybe even Twisted forward. Having that done before PyCon would be a *great* achievement. >>>>> >>>>> As a teaser, Twisted trunk + PyOpenSSL master + cryptography master = https://gist.github.com/hynek/2f0acd65a34523028168 >>>> I've just been reminded that some of the required initialization APIs >>>> are not in 0.1 so pyOpenSSL 0.14 will need cryptography 0.2. >>> Which specific APIs are those? >> >> They *were* ERR_load_crypto_strings() and ERR_load_SSL_strings >> but reaperhulk pointed out that cryptography 0.1 >> has SSL_load_error_strings so I've adjusted pyOpenSSL to use that >> instead. So now it again appears as though pyOpenSSL 0.14 will >> be satisfied with cryptography 0.1. > > Thanks. I was asking mostly for my own edification, but good to > know they were in fact wrapped :-). > > -glyph > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > > > > > -- > "I disapprove of what you say, but I will defend to the death your > right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From alexs at prol.etari.at Mon Jan 20 17:07:29 2014 From: alexs at prol.etari.at (alexs) Date: Mon, 20 Jan 2014 16:07:29 +0000 Subject: [Cryptography-dev] Thread safety in the OpenSSL backend Message-ID: <020239831f124b5267415eed72256106@prol.etari.at> TLDR; We would like to merge a thread safety solution for the OpenSSL bindings [1]. Probably this https://github.com/pyca/cryptography/pull/377 What do we need to do to get it reviewed and in master? As has been recently discovered by Hynek [2], initialising OpenSSL is more difficult than it should be. This is complicated by the Python VM doing its own initialisation of OpenSSL things inside the _ssl module. Normally this works in our favour as Python happens to call the same functions as us. If we have linked cryptography to a different version of OpenSSL to the one Python was compiled against (such as with bottled OpenSSL) however this can result in _ssl not doing any initialisation for us. In particular, it wont setup the threading callback required to make (among other things) the OpenSSL RNG thread safe. This is particularly annoying on OS X which ships with OpenSSL 0.9.8 still. There are a few approaches to this. 1. Ignore it, only support Python OpenSSL, document that somewhere. Will probably make OS X users sad. 2. Expose the callback setup functions and implement locking entirely in Python. This has 4-8x more overhead than the C version but is a small amount of code, relatively easy to test and works cross-platform. 3. Implement our own cross-platform locking wrappers and point OpenSSL at those. This is fast but is harder to test well while we have no Windows build infrastructure. Also puts even more code in C which we have no coverage data on. I have written a PR that implements option 3: https://github.com/pyca/cryptography/pull/377 The code was mostly borrowed from PyPy, which is mostly borrowed from an old version of CPython. It seems to solve my small crashing test cases on POSIX. We could really use some people who feel confident (or perhaps even competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this. It's not had a lot of eyes on and I'm particularly uncomfortable with the Windows parts since I lack the hardware (or software licenses) to test them. [1] https://github.com/pyca/cryptography/issues/330 [2] https://github.com/pyca/pyopenssl/issues/7 From alexs at prol.etari.at Mon Jan 20 17:10:00 2014 From: alexs at prol.etari.at (alexs) Date: Mon, 20 Jan 2014 16:10:00 +0000 Subject: [Cryptography-dev] Thread safety in the OpenSSL backend In-Reply-To: <020239831f124b5267415eed72256106@prol.etari.at> References: <020239831f124b5267415eed72256106@prol.etari.at> Message-ID: *facepalm* This is the correct PR https://github.com/pyca/cryptography/pull/352 On 20.01.2014 16:07, alexs wrote: > TLDR; We would like to merge a thread safety solution for the OpenSSL > bindings [1]. > > Probably this https://github.com/pyca/cryptography/pull/377 > > What do we need to do to get it reviewed and in master? > > As has been recently discovered by Hynek [2], initialising OpenSSL is > more difficult than it should be. This is complicated by the Python > VM > doing its own initialisation of OpenSSL things inside the _ssl > module. > Normally this works in our favour as Python happens to call the same > functions as us. If we have linked cryptography to a different > version > of OpenSSL to the one Python was compiled against (such as with > bottled OpenSSL) however this can result in _ssl not doing any > initialisation for us. In particular, it wont setup the threading > callback required to make (among other things) the OpenSSL RNG thread > safe. This is particularly annoying on OS X which ships with OpenSSL > 0.9.8 still. > > There are a few approaches to this. > > 1. Ignore it, only support Python OpenSSL, document that somewhere. > Will probably make OS X users sad. > 2. Expose the callback setup functions and implement locking > entirely in Python. This has 4-8x more overhead than the C version > but > is a small amount of code, relatively easy to test and works > cross-platform. > 3. Implement our own cross-platform locking wrappers and point > OpenSSL at those. This is fast but is harder to test well while we > have no Windows build infrastructure. Also puts even more code in C > which we have no coverage data on. > > I have written a PR that implements option 3: > https://github.com/pyca/cryptography/pull/377 > > The code was mostly borrowed from PyPy, which is mostly borrowed from > an old version of CPython. It seems to solve my small crashing test > cases on POSIX. > > We could really use some people who feel confident (or perhaps even > competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this. > It's not had a lot of eyes on and I'm particularly uncomfortable with > the Windows parts since I lack the hardware (or software licenses) to > test them. > > [1] https://github.com/pyca/cryptography/issues/330 > [2] https://github.com/pyca/pyopenssl/issues/7 > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev From alex.gaynor at gmail.com Mon Jan 20 17:10:44 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 20 Jan 2014 10:10:44 -0600 Subject: [Cryptography-dev] Thread safety in the OpenSSL backend In-Reply-To: <020239831f124b5267415eed72256106@prol.etari.at> References: <020239831f124b5267415eed72256106@prol.etari.at> Message-ID: How much would you hate me if I asked about doing a prototype of #2? It's the kind of performance issue that we can fix later very easily, but makes it much easier to review now. In general +1 on us fixing this, one suggestion though: this shouldn't be done in backend, it should be done somewhere ~~generic~~ so that pyOpenSSL can just call this function without needing to instantiating a backend. Alex On Mon, Jan 20, 2014 at 10:07 AM, alexs wrote: > TLDR; We would like to merge a thread safety solution for the OpenSSL > bindings [1]. > > Probably this https://github.com/pyca/cryptography/pull/377 > > What do we need to do to get it reviewed and in master? > > As has been recently discovered by Hynek [2], initialising OpenSSL is more > difficult than it should be. This is complicated by the Python VM doing its > own initialisation of OpenSSL things inside the _ssl module. Normally this > works in our favour as Python happens to call the same functions as us. If > we have linked cryptography to a different version of OpenSSL to the one > Python was compiled against (such as with bottled OpenSSL) however this can > result in _ssl not doing any initialisation for us. In particular, it wont > setup the threading callback required to make (among other things) the > OpenSSL RNG thread safe. This is particularly annoying on OS X which ships > with OpenSSL 0.9.8 still. > > There are a few approaches to this. > > 1. Ignore it, only support Python OpenSSL, document that somewhere. Will > probably make OS X users sad. > 2. Expose the callback setup functions and implement locking entirely in > Python. This has 4-8x more overhead than the C version but is a small > amount of code, relatively easy to test and works cross-platform. > 3. Implement our own cross-platform locking wrappers and point OpenSSL at > those. This is fast but is harder to test well while we have no Windows > build infrastructure. Also puts even more code in C which we have no > coverage data on. > > I have written a PR that implements option 3: https://github.com/pyca/ > cryptography/pull/377 > > The code was mostly borrowed from PyPy, which is mostly borrowed from an > old version of CPython. It seems to solve my small crashing test cases on > POSIX. > > We could really use some people who feel confident (or perhaps even > competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this. It's > not had a lot of eyes on and I'm particularly uncomfortable with the > Windows parts since I lack the hardware (or software licenses) to test them. > > [1] https://github.com/pyca/cryptography/issues/330 > [2] https://github.com/pyca/pyopenssl/issues/7 > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.l.kehrer at gmail.com Mon Jan 20 18:36:02 2014 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Mon, 20 Jan 2014 11:36:02 -0600 Subject: [Cryptography-dev] OpenSSL Random Engine PR Message-ID: Is anyone aware of blockers around the urandom engine PR (https://github.com/pyca/cryptography/pull/377) at this point? We?ve solicited feedback from as many people as we can and it appears this approach is generally considered to be a good one. What do we need to do to get this landed? For those desiring a refresher, OpenSSL has a CSPRNG that it seeds when starting up. Unfortunately, this state is then replicated if the process is forked and child processes can deliver similar/identical random values[1]. This issue came up for CPython (http://bugs.python.org/issue18747), but the solution of reseeding on pthread_atfork turns out to be dangerous and was backed out. The solution we?ve come up with is to build an OpenSSL engine that replaces the standard random source with one that fetches entropy from /dev/urandom (or CryptGenRandom on Windows, which is Fortuna). Advantages to this approach: * Fixes fork issue * Gets entropy from a source many (most?) cryptographers consider less likely to have serious issues than a userland RNG. Disadvantages: * Slower than OpenSSL?s native generation (this doesn?t seem to be an issue, but is worth noting) * Affects global OpenSSL state The last one means that when activating this engine anything in the same process that is using OpenSSL will switch to this engine. That includes Python?s own SSL module (unless cryptography is built against an alternate OpenSSL). The current codebase registers the engine in the binding, but does not activate it. The backend activates it. This means consumers of the binding only (e.g. PyOpenSSL) will not use the engine unless they explicitly opt-in. [1] OpenSSL has put in some mitigation for this problem in trunk (not sure if it was released with 1.0.1f), but we can?t rely on that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dreid at dreid.org Mon Jan 20 18:41:12 2014 From: dreid at dreid.org (David Reid) Date: Mon, 20 Jan 2014 09:41:12 -0800 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: References: Message-ID: "- [ ] documentation? ^ - blocker --? David Reid From:?Paul Kehrer Paul Kehrer Reply:?cryptography-dev at python.org cryptography-dev at python.org Date:?January 20, 2014 at 9:37:54 AM To:?cryptography-dev at python.org cryptography-dev at python.org Subject:? [Cryptography-dev] OpenSSL Random Engine PR Is anyone aware of blockers around the urandom engine PR (https://github.com/pyca/cryptography/pull/377) at this point? We?ve solicited feedback from as many people as we can and it appears this approach is generally considered to be a good one. What do we need to do to get this landed? For those desiring a refresher, OpenSSL has a CSPRNG that it seeds when starting up. Unfortunately, this state is then replicated if the process is forked and child processes can deliver similar/identical random values[1]. This issue came up for CPython (http://bugs.python.org/issue18747), but the solution of reseeding on pthread_atfork turns out to be dangerous and was backed out. The solution we?ve come up with is to build an OpenSSL engine that replaces the standard random source with one that fetches entropy from /dev/urandom (or CryptGenRandom on Windows, which is Fortuna). Advantages to this approach: * Fixes fork issue * Gets entropy from a source many (most?) cryptographers consider less likely to have serious issues than a userland RNG. Disadvantages: * Slower than OpenSSL?s native generation (this doesn?t seem to be an issue, but is worth noting) * Affects global OpenSSL state The last one means that when activating this engine anything in the same process that is using OpenSSL will switch to this engine. That includes Python?s own SSL module (unless cryptography is built against an alternate OpenSSL). The current codebase registers the engine in the binding, but does not activate it. The backend activates it. This means consumers of the binding only (e.g. PyOpenSSL) will not use the engine unless they explicitly opt-in. [1] OpenSSL has put in some mitigation for this problem in trunk (not sure if it was released with 1.0.1f), but we can?t rely on that. _______________________________________________ Cryptography-dev mailing list Cryptography-dev at python.org https://mail.python.org/mailman/listinfo/cryptography-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexs at prol.etari.at Mon Jan 20 19:25:13 2014 From: alexs at prol.etari.at (Alex Stapleton) Date: Mon, 20 Jan 2014 18:25:13 +0000 Subject: [Cryptography-dev] Thread safety in the OpenSSL backend In-Reply-To: References: <020239831f124b5267415eed72256106@prol.etari.at> Message-ID: <143b0e60c18.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> I've already implemented 2 as well so its not too awful :-) Needs tests and things though. The current PR is all exposed on the binding so PyOpenSSL could easily use it. Where do you suggest a Python implementation should live? A utils module in the binding.openssl package? I failed to mention option 4 in my previous post too. 4. Enable the GIL on all of the OpenSSL bindings. Not entirely sure how you convince cffi to do that though. Might not be possible? Sent with AquaMail for Android http://www.aqua-mail.com On 20 January 2014 16:10:44 Alex Gaynor wrote: > How much would you hate me if I asked about doing a prototype of #2? It's > the kind of performance issue that we can fix later very easily, but makes > it much easier to review now. > > In general +1 on us fixing this, one suggestion though: this shouldn't be > done in backend, it should be done somewhere ~~generic~~ so that pyOpenSSL > can just call this function without needing to instantiating a backend. > > Alex > > > On Mon, Jan 20, 2014 at 10:07 AM, alexs wrote: > > > TLDR; We would like to merge a thread safety solution for the OpenSSL > > bindings [1]. > > > > Probably this https://github.com/pyca/cryptography/pull/377 > > > > What do we need to do to get it reviewed and in master? > > > > As has been recently discovered by Hynek [2], initialising OpenSSL is more > > difficult than it should be. This is complicated by the Python VM doing its > > own initialisation of OpenSSL things inside the _ssl module. Normally this > > works in our favour as Python happens to call the same functions as us. If > > we have linked cryptography to a different version of OpenSSL to the one > > Python was compiled against (such as with bottled OpenSSL) however this can > > result in _ssl not doing any initialisation for us. In particular, it wont > > setup the threading callback required to make (among other things) the > > OpenSSL RNG thread safe. This is particularly annoying on OS X which ships > > with OpenSSL 0.9.8 still. > > > > There are a few approaches to this. > > > > 1. Ignore it, only support Python OpenSSL, document that somewhere. Will > > probably make OS X users sad. > > 2. Expose the callback setup functions and implement locking entirely in > > Python. This has 4-8x more overhead than the C version but is a small > > amount of code, relatively easy to test and works cross-platform. > > 3. Implement our own cross-platform locking wrappers and point OpenSSL at > > those. This is fast but is harder to test well while we have no Windows > > build infrastructure. Also puts even more code in C which we have no > > coverage data on. > > > > I have written a PR that implements option 3: https://github.com/pyca/ > > cryptography/pull/377 > > > > The code was mostly borrowed from PyPy, which is mostly borrowed from an > > old version of CPython. It seems to solve my small crashing test cases on > > POSIX. > > > > We could really use some people who feel confident (or perhaps even > > competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this. It's > > not had a lot of eyes on and I'm particularly uncomfortable with the > > Windows parts since I lack the hardware (or software licenses) to test them. > > > > [1] https://github.com/pyca/cryptography/issues/330 > > [2] https://github.com/pyca/pyopenssl/issues/7 > > _______________________________________________ > > Cryptography-dev mailing list > > Cryptography-dev at python.org > > https://mail.python.org/mailman/listinfo/cryptography-dev > > > > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Mon Jan 20 19:26:13 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 20 Jan 2014 12:26:13 -0600 Subject: [Cryptography-dev] Thread safety in the OpenSSL backend In-Reply-To: <143b0e60c18.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> References: <020239831f124b5267415eed72256106@prol.etari.at> <143b0e60c18.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: Maybe a method on `Binding` is the right place? Not sure, let's get the code up, then we can find a place. What do you mean "Enable the GIL"? Alex On Mon, Jan 20, 2014 at 12:25 PM, Alex Stapleton wrote: > I've already implemented 2 as well so its not too awful :-) Needs tests > and things though. > > The current PR is all exposed on the binding so PyOpenSSL could easily use > it. Where do you suggest a Python implementation should live? A utils > module in the binding.openssl package? > > I failed to mention option 4 in my previous post too. > > 4. Enable the GIL on all of the OpenSSL bindings. > > Not entirely sure how you convince cffi to do that though. Might not be > possible? > > Sent with AquaMail for Android > http://www.aqua-mail.com > > On 20 January 2014 16:10:44 Alex Gaynor wrote: > > How much would you hate me if I asked about doing a prototype of #2? It's > the kind of performance issue that we can fix later very easily, but makes > it much easier to review now. > > In general +1 on us fixing this, one suggestion though: this shouldn't be > done in backend, it should be done somewhere ~~generic~~ so that pyOpenSSL > can just call this function without needing to instantiating a backend. > > Alex > > > On Mon, Jan 20, 2014 at 10:07 AM, alexs wrote: > >> TLDR; We would like to merge a thread safety solution for the OpenSSL >> bindings [1]. >> >> Probably this https://github.com/pyca/cryptography/pull/377 >> >> What do we need to do to get it reviewed and in master? >> >> As has been recently discovered by Hynek [2], initialising OpenSSL is >> more difficult than it should be. This is complicated by the Python VM >> doing its own initialisation of OpenSSL things inside the _ssl module. >> Normally this works in our favour as Python happens to call the same >> functions as us. If we have linked cryptography to a different version of >> OpenSSL to the one Python was compiled against (such as with bottled >> OpenSSL) however this can result in _ssl not doing any initialisation for >> us. In particular, it wont setup the threading callback required to make >> (among other things) the OpenSSL RNG thread safe. This is particularly >> annoying on OS X which ships with OpenSSL 0.9.8 still. >> >> There are a few approaches to this. >> >> 1. Ignore it, only support Python OpenSSL, document that somewhere. Will >> probably make OS X users sad. >> 2. Expose the callback setup functions and implement locking entirely in >> Python. This has 4-8x more overhead than the C version but is a small >> amount of code, relatively easy to test and works cross-platform. >> 3. Implement our own cross-platform locking wrappers and point OpenSSL >> at those. This is fast but is harder to test well while we have no Windows >> build infrastructure. Also puts even more code in C which we have no >> coverage data on. >> >> I have written a PR that implements option 3: https://github.com/pyca/ >> cryptography/pull/377 >> >> The code was mostly borrowed from PyPy, which is mostly borrowed from an >> old version of CPython. It seems to solve my small crashing test cases on >> POSIX. >> >> We could really use some people who feel confident (or perhaps even >> competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this. >> It's not had a lot of eyes on and I'm particularly uncomfortable with the >> Windows parts since I lack the hardware (or software licenses) to test them. >> >> [1] https://github.com/pyca/cryptography/issues/330 >> [2] https://github.com/pyca/pyopenssl/issues/7 >> _______________________________________________ >> Cryptography-dev mailing list >> Cryptography-dev at python.org >> https://mail.python.org/mailman/listinfo/cryptography-dev >> > > > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexs at prol.etari.at Mon Jan 20 19:36:14 2014 From: alexs at prol.etari.at (Alex Stapleton) Date: Mon, 20 Jan 2014 18:36:14 +0000 Subject: [Cryptography-dev] Thread safety in the OpenSSL backend In-Reply-To: References: <020239831f124b5267415eed72256106@prol.etari.at> <143b0e60c18.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: <143b0eef170.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> AIUI the cffi bindings release the GIL during the C call. This allows another thread to run that can then end up calling into OpenSSL again. If we leave the GIL locked during the call there's no way for there to be concurrent calls. Gives you 0 parallelism on all operations though. Sent with AquaMail for Android http://www.aqua-mail.com On 20 January 2014 18:26:13 Alex Gaynor wrote: > Maybe a method on `Binding` is the right place? Not sure, let's get the > code up, then we can find a place. > > What do you mean "Enable the GIL"? > > Alex > > > On Mon, Jan 20, 2014 at 12:25 PM, Alex Stapleton wrote: > > > I've already implemented 2 as well so its not too awful :-) Needs tests > > and things though. > > > > The current PR is all exposed on the binding so PyOpenSSL could easily use > > it. Where do you suggest a Python implementation should live? A utils > > module in the binding.openssl package? > > > > I failed to mention option 4 in my previous post too. > > > > 4. Enable the GIL on all of the OpenSSL bindings. > > > > Not entirely sure how you convince cffi to do that though. Might not be > > possible? > > > > Sent with AquaMail for Android > > http://www.aqua-mail.com > > > > On 20 January 2014 16:10:44 Alex Gaynor wrote: > > > > How much would you hate me if I asked about doing a prototype of #2? It's > > the kind of performance issue that we can fix later very easily, but makes > > it much easier to review now. > > > > In general +1 on us fixing this, one suggestion though: this shouldn't be > > done in backend, it should be done somewhere ~~generic~~ so that pyOpenSSL > > can just call this function without needing to instantiating a backend. > > > > Alex > > > > > > On Mon, Jan 20, 2014 at 10:07 AM, alexs wrote: > > > >> TLDR; We would like to merge a thread safety solution for the OpenSSL > >> bindings [1]. > >> > >> Probably this https://github.com/pyca/cryptography/pull/377 > >> > >> What do we need to do to get it reviewed and in master? > >> > >> As has been recently discovered by Hynek [2], initialising OpenSSL is > >> more difficult than it should be. This is complicated by the Python VM > >> doing its own initialisation of OpenSSL things inside the _ssl module. > >> Normally this works in our favour as Python happens to call the same > >> functions as us. If we have linked cryptography to a different version of > >> OpenSSL to the one Python was compiled against (such as with bottled > >> OpenSSL) however this can result in _ssl not doing any initialisation for > >> us. In particular, it wont setup the threading callback required to make > >> (among other things) the OpenSSL RNG thread safe. This is particularly > >> annoying on OS X which ships with OpenSSL 0.9.8 still. > >> > >> There are a few approaches to this. > >> > >> 1. Ignore it, only support Python OpenSSL, document that somewhere. Will > >> probably make OS X users sad. > >> 2. Expose the callback setup functions and implement locking entirely in > >> Python. This has 4-8x more overhead than the C version but is a small > >> amount of code, relatively easy to test and works cross-platform. > >> 3. Implement our own cross-platform locking wrappers and point OpenSSL > >> at those. This is fast but is harder to test well while we have no Windows > >> build infrastructure. Also puts even more code in C which we have no > >> coverage data on. > >> > >> I have written a PR that implements option 3: https://github.com/pyca/ > >> cryptography/pull/377 > >> > >> The code was mostly borrowed from PyPy, which is mostly borrowed from an > >> old version of CPython. It seems to solve my small crashing test cases on > >> POSIX. > >> > >> We could really use some people who feel confident (or perhaps even > >> competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this. > >> It's not had a lot of eyes on and I'm particularly uncomfortable with the > >> Windows parts since I lack the hardware (or software licenses) to test them. > >> > >> [1] https://github.com/pyca/cryptography/issues/330 > >> [2] https://github.com/pyca/pyopenssl/issues/7 > >> _______________________________________________ > >> Cryptography-dev mailing list > >> Cryptography-dev at python.org > >> https://mail.python.org/mailman/listinfo/cryptography-dev > >> > > > > > > > > -- > > "I disapprove of what you say, but I will defend to the death your right > > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > > "The people's good is the highest law." -- Cicero > > GPG Key fingerprint: 125F 5C67 DFE9 4084 > > > > > > _______________________________________________ > > Cryptography-dev mailing list > > Cryptography-dev at python.org > > https://mail.python.org/mailman/listinfo/cryptography-dev > > > > > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gaynor at gmail.com Mon Jan 20 19:37:11 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 20 Jan 2014 12:37:11 -0600 Subject: [Cryptography-dev] Thread safety in the OpenSSL backend In-Reply-To: <143b0eef170.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> References: <020239831f124b5267415eed72256106@prol.etari.at> <143b0e60c18.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> <143b0eef170.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: Yeah, seems like a bad idea, people definitely expect something like read() on an SSL socket to release the GIL. Alex On Mon, Jan 20, 2014 at 12:36 PM, Alex Stapleton wrote: > AIUI the cffi bindings release the GIL during the C call. This allows > another thread to run that can then end up calling into OpenSSL again. If > we leave the GIL locked during the call there's no way for there to be > concurrent calls. Gives you 0 parallelism on all operations though. > > Sent with AquaMail for Android > http://www.aqua-mail.com > > On 20 January 2014 18:26:13 Alex Gaynor wrote: > > Maybe a method on `Binding` is the right place? Not sure, let's get the > code up, then we can find a place. > > What do you mean "Enable the GIL"? > > Alex > > > On Mon, Jan 20, 2014 at 12:25 PM, Alex Stapleton wrote: > >> I've already implemented 2 as well so its not too awful :-) Needs >> tests and things though. >> >> The current PR is all exposed on the binding so PyOpenSSL could easily >> use it. Where do you suggest a Python implementation should live? A utils >> module in the binding.openssl package? >> >> I failed to mention option 4 in my previous post too. >> >> 4. Enable the GIL on all of the OpenSSL bindings. >> >> Not entirely sure how you convince cffi to do that though. Might not be >> possible? >> >> Sent with AquaMail for Android >> http://www.aqua-mail.com >> >> On 20 January 2014 16:10:44 Alex Gaynor wrote: >> >> How much would you hate me if I asked about doing a prototype of #2? It's >> the kind of performance issue that we can fix later very easily, but makes >> it much easier to review now. >> >> In general +1 on us fixing this, one suggestion though: this shouldn't be >> done in backend, it should be done somewhere ~~generic~~ so that pyOpenSSL >> can just call this function without needing to instantiating a backend. >> >> Alex >> >> >> On Mon, Jan 20, 2014 at 10:07 AM, alexs wrote: >> >>> TLDR; We would like to merge a thread safety solution for the OpenSSL >>> bindings [1]. >>> >>> Probably this https://github.com/pyca/cryptography/pull/377 >>> >>> What do we need to do to get it reviewed and in master? >>> >>> As has been recently discovered by Hynek [2], initialising OpenSSL is >>> more difficult than it should be. This is complicated by the Python VM >>> doing its own initialisation of OpenSSL things inside the _ssl module. >>> Normally this works in our favour as Python happens to call the same >>> functions as us. If we have linked cryptography to a different version of >>> OpenSSL to the one Python was compiled against (such as with bottled >>> OpenSSL) however this can result in _ssl not doing any initialisation for >>> us. In particular, it wont setup the threading callback required to make >>> (among other things) the OpenSSL RNG thread safe. This is particularly >>> annoying on OS X which ships with OpenSSL 0.9.8 still. >>> >>> There are a few approaches to this. >>> >>> 1. Ignore it, only support Python OpenSSL, document that somewhere. >>> Will probably make OS X users sad. >>> 2. Expose the callback setup functions and implement locking entirely >>> in Python. This has 4-8x more overhead than the C version but is a small >>> amount of code, relatively easy to test and works cross-platform. >>> 3. Implement our own cross-platform locking wrappers and point OpenSSL >>> at those. This is fast but is harder to test well while we have no Windows >>> build infrastructure. Also puts even more code in C which we have no >>> coverage data on. >>> >>> I have written a PR that implements option 3: https://github.com/pyca/ >>> cryptography/pull/377 >>> >>> The code was mostly borrowed from PyPy, which is mostly borrowed from an >>> old version of CPython. It seems to solve my small crashing test cases on >>> POSIX. >>> >>> We could really use some people who feel confident (or perhaps even >>> competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this. >>> It's not had a lot of eyes on and I'm particularly uncomfortable with the >>> Windows parts since I lack the hardware (or software licenses) to test them. >>> >>> [1] https://github.com/pyca/cryptography/issues/330 >>> [2] https://github.com/pyca/pyopenssl/issues/7 >>> _______________________________________________ >>> Cryptography-dev mailing list >>> Cryptography-dev at python.org >>> https://mail.python.org/mailman/listinfo/cryptography-dev >>> >> >> >> >> -- >> "I disapprove of what you say, but I will defend to the death your right >> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) >> "The people's good is the highest law." -- Cicero >> GPG Key fingerprint: 125F 5C67 DFE9 4084 >> >> >> _______________________________________________ >> Cryptography-dev mailing list >> Cryptography-dev at python.org >> https://mail.python.org/mailman/listinfo/cryptography-dev >> >> > > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-paul at hybridcluster.com Mon Jan 20 20:26:39 2014 From: jean-paul at hybridcluster.com (Jean-Paul Calderone) Date: Mon, 20 Jan 2014 14:26:39 -0500 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: References: Message-ID: <52DD786F.5060802@hybridcluster.com> On 01/20/2014 12:36 PM, Paul Kehrer wrote: > Is anyone aware of blockers around the urandom engine PR > (https://github.com/pyca/cryptography/pull/377) at this point? We've > solicited feedback from as many people as we can and it appears this > approach is generally considered to be a good one. What do we need to > do to get this landed? > > > For those desiring a refresher, OpenSSL has a CSPRNG that it seeds > when starting up. Unfortunately, this state is then replicated if the > process is forked and child processes can deliver similar/identical > random values[1]. This issue came up for CPython > (http://bugs.python.org/issue18747), but the solution of reseeding on > pthread_atfork turns out to be dangerous and was backed out. > > The solution we've come up with is to build an OpenSSL engine that > replaces the standard random source with one that fetches entropy from > /dev/urandom (or CryptGenRandom on Windows, which is Fortuna). Won't this make OpenSSL suck up all entropy on a system fairly easily? For example, every SSL handshake requires some random bytes (I could count them I guess but it doesn't really matter). Consider an HTTPS server accepting hundreds of requests per second... Since the idea is to use urandom, this won't cause a problem as obvious as blocking the SSL server waiting for more entropy. It will exhaust what the system (at least on Linux, I don't know much about urandom on other platforms) considers the actual amount of available entropy. This may mean that any other process that really wants /dev/random may be unable to operate. Jean-Paul > > Advantages to this approach: > * Fixes fork issue > * Gets entropy from a source many (most?) cryptographers consider less > likely to have serious issues than a userland RNG. > > Disadvantages: > * Slower than OpenSSL's native generation (this doesn't seem to be an > issue, but is worth noting) > * Affects global OpenSSL state > > The last one means that when activating this engine anything in the > same process that is using OpenSSL will switch to this engine. That > includes Python's own SSL module (unless cryptography is built against > an alternate OpenSSL). > > The current codebase registers the engine in the binding, but does not > activate it. The backend activates it. This means consumers of the > binding only (e.g. PyOpenSSL) will not use the engine unless they > explicitly opt-in. > > [1] OpenSSL has put in some mitigation for this problem in trunk (not > sure if it was released with 1.0.1f), but we can't rely on that. > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From paul.l.kehrer at gmail.com Mon Jan 20 21:51:40 2014 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Mon, 20 Jan 2014 14:51:40 -0600 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <52DD786F.5060802@hybridcluster.com> References: <52DD786F.5060802@hybridcluster.com> Message-ID: <4D8B80093F5E488E88307C148C809B3B@gmail.com> On Monday, January 20, 2014 at 1:26 PM, Jean-Paul Calderone wrote: > On 01/20/2014 12:36 PM, Paul Kehrer wrote: > > Is anyone aware of blockers around the urandom engine PR (https://github.com/pyca/cryptography/pull/377) at this point? We?ve solicited feedback from as many people as we can and it appears this approach is generally considered to be a good one. What do we need to do to get this landed? > > > > > > For those desiring a refresher, OpenSSL has a CSPRNG that it seeds when starting up. Unfortunately, this state is then replicated if the process is forked and child processes can deliver similar/identical random values[1]. This issue came up for CPython (http://bugs.python.org/issue18747), but the solution of reseeding on pthread_atfork turns out to be dangerous and was backed out. > > > > The solution we?ve come up with is to build an OpenSSL engine that replaces the standard random source with one that fetches entropy from /dev/urandom (or CryptGenRandom on Windows, which is Fortuna). > Won't this make OpenSSL suck up all entropy on a system fairly easily? For example, every SSL handshake requires some random bytes (I could count them I guess but it doesn't really matter). Consider an HTTPS server accepting hundreds of requests per second... > > Since the idea is to use urandom, this won't cause a problem as obvious as blocking the SSL server waiting for more entropy. It will exhaust what the system (at least on Linux, I don't know much about urandom on other platforms) considers the actual amount of available entropy. This may mean that any other process that really wants /dev/random may be unable to operate. > > Jean-Paul I don?t know under what circumstances /dev/urandom reseeds itself on Linux, so I can't answer this question. Greg Price can probably give us an answer as he?s involved with improving the kernel?s RNG. Anyone know if he?s on this list or if we need to ping him on IRC? > > > > > > Advantages to this approach: > > * Fixes fork issue > > * Gets entropy from a source many (most?) cryptographers consider less likely to have serious issues than a userland RNG. > > > > Disadvantages: > > * Slower than OpenSSL?s native generation (this doesn?t seem to be an issue, but is worth noting) > > * Affects global OpenSSL state > > > > The last one means that when activating this engine anything in the same process that is using OpenSSL will switch to this engine. That includes Python?s own SSL module (unless cryptography is built against an alternate OpenSSL). > > > > The current codebase registers the engine in the binding, but does not activate it. The backend activates it. This means consumers of the binding only (e.g. PyOpenSSL) will not use the engine unless they explicitly opt-in. > > > > [1] OpenSSL has put in some mitigation for this problem in trunk (not sure if it was released with 1.0.1f), but we can?t rely on that. > > > > _______________________________________________ Cryptography-dev mailing list Cryptography-dev at python.org (mailto:Cryptography-dev at python.org) https://mail.python.org/mailman/listinfo/cryptography-dev > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org (mailto:Cryptography-dev at python.org) > https://mail.python.org/mailman/listinfo/cryptography-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at lvh.io Mon Jan 20 21:54:36 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 20 Jan 2014 21:54:36 +0100 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <52DD786F.5060802@hybridcluster.com> References: <52DD786F.5060802@hybridcluster.com> Message-ID: On Jan 20, 2014 9:11 PM, "Jean-Paul Calderone" wrote: > > On 01/20/2014 12:36 PM, Paul Kehrer wrote: >> >> Is anyone aware of blockers around the urandom engine PR ( https://github.com/pyca/cryptography/pull/377) at this point? We?ve solicited feedback from as many people as we can and it appears this approach is generally considered to be a good one. What do we need to do to get this landed? >> >> >> For those desiring a refresher, OpenSSL has a CSPRNG that it seeds when starting up. Unfortunately, this state is then replicated if the process is forked and child processes can deliver similar/identical random values[1]. This issue came up for CPython (http://bugs.python.org/issue18747), but the solution of reseeding on pthread_atfork turns out to be dangerous and was backed out. >> >> The solution we?ve come up with is to build an OpenSSL engine that replaces the standard random source with one that fetches entropy from /dev/urandom (or CryptGenRandom on Windows, which is Fortuna). > > > Won't this make OpenSSL suck up all entropy on a system fairly easily? For example, every SSL handshake requires some random bytes (I could count them I guess but it doesn't really matter). Consider an HTTPS server accepting hundreds of requests per second... On phone, so brief, but: no. > Since the idea is to use urandom, this won't cause a problem as obvious as blocking the SSL server waiting for more entropy. It will exhaust what the system (at least on Linux, I don't know much about urandom on other platforms) considers the actual amount of available entropy. This may mean that any other process that really wants /dev/random may be unable to operate. > > Jean-Paul > > >> >> Advantages to this approach: >> * Fixes fork issue >> * Gets entropy from a source many (most?) cryptographers consider less likely to have serious issues than a userland RNG. >> >> Disadvantages: >> * Slower than OpenSSL?s native generation (this doesn?t seem to be an issue, but is worth noting) >> * Affects global OpenSSL state >> >> The last one means that when activating this engine anything in the same process that is using OpenSSL will switch to this engine. That includes Python?s own SSL module (unless cryptography is built against an alternate OpenSSL). >> >> The current codebase registers the engine in the binding, but does not activate it. The backend activates it. This means consumers of the binding only (e.g. PyOpenSSL) will not use the engine unless they explicitly opt-in. >> >> [1] OpenSSL has put in some mitigation for this problem in trunk (not sure if it was released with 1.0.1f), but we can?t rely on that. >> >> >> _______________________________________________ >> Cryptography-dev mailing list >> Cryptography-dev at python.org >> https://mail.python.org/mailman/listinfo/cryptography-dev > > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul at mcmillan.ws Mon Jan 20 21:49:40 2014 From: paul at mcmillan.ws (Paul McMillan) Date: Mon, 20 Jan 2014 20:49:40 +0000 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <52DD786F.5060802@hybridcluster.com> References: <52DD786F.5060802@hybridcluster.com> Message-ID: > Since the idea is to use urandom, this won't cause a problem as obvious as > blocking the SSL server waiting for more entropy. It will exhaust what the > system (at least on Linux, I don't know much about urandom on other > platforms) considers the actual amount of available entropy. This may mean > that any other process that really wants /dev/random may be unable to > operate. I don't think this is something that we need to worry about. ANY process that depends on /dev/random may block for an indefinite period of time, and doing so is (IMO) always bad design, regardless of what certain obstinate people may claim. Having an RNG that interacts with the system entropy pool (and uses it when available) is much better than a userland generator that starts with a seed and goes from there (which is how we got into this pickle in the first place). This also makes it easier for end-users to make use of hardware RNG features. -Paul From jean-paul at hybridcluster.com Mon Jan 20 22:30:37 2014 From: jean-paul at hybridcluster.com (Jean-Paul Calderone) Date: Mon, 20 Jan 2014 16:30:37 -0500 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: References: <52DD786F.5060802@hybridcluster.com> Message-ID: <52DD957D.5080901@hybridcluster.com> On 01/20/2014 03:54 PM, Laurens Van Houtven wrote: > On phone, so brief, but: no. It's email... We can wait for you to get off the phone so you can type a complete response. :) Jean-Paul -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From alexs at prol.etari.at Mon Jan 20 22:42:02 2014 From: alexs at prol.etari.at (Alex Stapleton) Date: Mon, 20 Jan 2014 21:42:02 +0000 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <4D8B80093F5E488E88307C148C809B3B@gmail.com> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> Message-ID: <143b198a6c0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> I'm not certain urandom ever reduces entropy_avail? The OpenSSL RNG also seeds from the kernel anyway so it can also cause depletion in some circumstances. Generally I don't think entropy estimation is our problem to fix. Separately it is probably worth documenting the entropy issue on Linux though just because it's generally useful info. Perhaps with some suggested solutions such as an entropy key, egd, havegd, etc Sent with AquaMail for Android http://www.aqua-mail.com On 20 January 2014 20:51:40 Paul Kehrer wrote: > On Monday, January 20, 2014 at 1:26 PM, Jean-Paul Calderone wrote: > > On 01/20/2014 12:36 PM, Paul Kehrer wrote: > Is anyone aware of blockers > around the urandom engine PR > (https://github.com/pyca/cryptography/pull/377) at this point? We?ve > solicited feedback from as many people as we can and it appears this > approach is generally considered to be a good one. What do we need to do to > get this landed? > > > For those desiring a refresher, OpenSSL has a CSPRNG > that it seeds when starting up. Unfortunately, this state is then > replicated if the process is forked and child processes can deliver > similar/identical random values[1]. This issue came up for CPython > (http://bugs.python.org/issue18747), but the solution of reseeding on > pthread_atfork turns out to be dangerous and was backed out. > > The > solution we?ve come up with is to build an OpenSSL engine that replaces the > standard random source with one that fetches entropy from /dev/urandom (or > CryptGenRandom on Windows, which is Fortuna). Won't this make OpenSSL suck > up all entropy on a system fairly easily? For example, every SSL handshake > requires some random bytes (I could count them I guess but it doesn't > really matter). Consider an HTTPS server accepting hundreds of requests > per second... > > Since the idea is to use urandom, this won't cause a problem as obvious > as blocking the SSL server waiting for more entropy. It will exhaust what > the system (at least on Linux, I don't know much about urandom on other > platforms) considers the actual amount of available entropy. This may mean > that any other process that really wants /dev/random may be unable to operate. > > Jean-Paul > I don?t know under what circumstances /dev/urandom reseeds itself on Linux, > so I can't answer this question. Greg Price can probably give us an answer > as he?s involved with improving the kernel?s RNG. Anyone know if he?s on > this list or if we need to ping him on IRC? > > > > Advantages to this approach: > * Fixes fork issue > > > * Gets entropy from a source many (most?) cryptographers consider less > likely to have serious issues than a userland RNG. > > > > Disadvantages: > * Slower than OpenSSL?s native generation (this > doesn?t seem to be an issue, but is worth noting) > > > * Affects global OpenSSL state > > > > The last one means that when activating this engine anything in the > same process that is using OpenSSL will switch to this engine. That > includes Python?s own SSL module (unless cryptography is built against an > alternate OpenSSL). > > The current codebase registers the engine in the > binding, but does not activate it. The backend activates it. This means > consumers of the binding only (e.g. PyOpenSSL) will not use the engine > unless they explicitly opt-in. > > [1] OpenSSL has put in some mitigation > for this problem in trunk (not sure if it was released with 1.0.1f), but we > can?t rely on that. > > _______________________________________________ > Cryptography-dev mailing list Cryptography-dev at python.org > (mailto:Cryptography-dev at python.org) > https://mail.python.org/mailman/listinfo/cryptography-dev > _______________________________________________ > > Cryptography-dev mailing list > > Cryptography-dev at python.org (mailto:Cryptography-dev at python.org) > > https://mail.python.org/mailman/listinfo/cryptography-dev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at lvh.io Mon Jan 20 22:58:03 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 20 Jan 2014 22:58:03 +0100 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <143b198a6c0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <143b198a6c0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: On Mon, Jan 20, 2014 at 10:42 PM, Alex Stapleton wrote: > I'm not certain urandom ever reduces entropy_avail? > I think it does (or at least shouldshould); extract_entropy_user calls account which does the accounting. > The OpenSSL RNG also seeds from the kernel anyway so it can also cause > depletion in some circumstances. Generally I don't think entropy estimation > is our problem to fix. > Absolutely. > Separately it is probably worth documenting the entropy issue on Linux > though just because it's generally useful info. Perhaps with some suggested > solutions such as an entropy key, egd, havegd, etc > Why is Linux special here? cheers lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Mon Jan 20 23:04:14 2014 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 20 Jan 2014 14:04:14 -0800 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <143b198a6c0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: <5159E022-D93B-4ABD-BB0F-00DA7C96DC13@twistedmatrix.com> On Jan 20, 2014, at 1:58 PM, Laurens Van Houtven <_ at lvh.io> wrote: > Separately it is probably worth documenting the entropy issue on Linux though just because it's generally useful info. Perhaps with some suggested solutions such as an entropy key, egd, havegd, etc > > Why is Linux special here? FreeBSD and OS X use Yarrow, and Windows uses Fortuna, so nobody but Linux even has an exhaustible / blocking random "real actual for real entropy" API. You can still exhaust the seed pool but as far as I know it would be literally impossible to tell (like, mathematically provably impossible by the properties of the various CSPRNGs used) that you had done so. On Linux, /dev/random might still block sometimes, so you can tell that way as a regular user, and your applications might get stuck. At least, as far as I know. The usual caveats about me not being a cryptographer, or actually knowing really anything about anything at all, apply. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4124 bytes Desc: not available URL: From _ at lvh.io Mon Jan 20 23:05:06 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 20 Jan 2014 23:05:06 +0100 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <52DD957D.5080901@hybridcluster.com> References: <52DD786F.5060802@hybridcluster.com> <52DD957D.5080901@hybridcluster.com> Message-ID: On Mon, Jan 20, 2014 at 10:30 PM, Jean-Paul Calderone < jean-paul at hybridcluster.com> wrote: > On 01/20/2014 03:54 PM, Laurens Van Houtven wrote: > > On phone, so brief, but: no. > > It's email... We can wait for you to get off the phone so you can type a > complete response. :) > Okay. The complete version isn't much longer. They're both just CSPRNGs; entropy starving can occur on both of them, but shouldn't in most cases for servers like you outlined (although it's gotten more probable now that spinning rust is getting rarer and rarer). There is at least no reason to assume that using the OpenSSL userspace one will lead to better results than using the urandom CSPRNG; otherwise the kernelspace one would just do that of course ;-) The main argument against the OpenSSL one is that it is *way* easier to screw it up, and the failure cases are typically catastrophic (like same CSPRNG state...). The only counterargument that I've heard is that using urandom might leak some people who already have local code execution (on the same logical machine) how much entropy you're using. That's true, but AFAIK that has never been turned into anything remotely threatening. So, yeah, just use urandom/CryptGenRandom. lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at lvh.io Mon Jan 20 23:09:39 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 20 Jan 2014 23:09:39 +0100 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <5159E022-D93B-4ABD-BB0F-00DA7C96DC13@twistedmatrix.com> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <143b198a6c0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> <5159E022-D93B-4ABD-BB0F-00DA7C96DC13@twistedmatrix.com> Message-ID: On Mon, Jan 20, 2014 at 11:04 PM, Glyph Lefkowitz wrote: > FreeBSD and OS X use Yarrow, and Windows uses Fortuna, so nobody but Linux > even has an exhaustible / blocking random "real actual for real entropy" > API. You can still exhaust the seed pool but as far as I know it would be > literally impossible to tell (like, mathematically provably impossible by > the properties of the various CSPRNGs used) that you had done so. > On Linux, /dev/random might still block sometimes, so you can tell that way > as a regular user, and your applications might get stuck. > > At least, as far as I know. The usual caveats about me not being a > cryptographer, or actually knowing really anything about anything at all, > apply. > > -glyph > > I'm familiar with the CSPRNGs in modern OSes; I just thought we were talking about exhausting the entropy of the CSPRNGs (so, say, urandom on Linux and (u)random on FreeBSD and CryptGenRandom on Windows). IIUC, none of the systems in play here actually use /dev/random in steady state (just for key generation maybe). If the grandparent indeed meant /dev/random vs /dev/urandom on Linux that would explain why, yes :-) (FYI, I've yet to hear a cryptographer whose opinion on the blocking /dev/random because better entropy in Linux isn't that it's pretty much hogwash. djb is convinced they should just give up trying and implement Fortuna or Yarrow already.) cheers lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexs at prol.etari.at Mon Jan 20 23:16:53 2014 From: alexs at prol.etari.at (Alex Stapleton) Date: Mon, 20 Jan 2014 22:16:53 +0000 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <143b198a6c0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> <5159E022-D93B-4ABD-BB0F-00DA7C96DC13@twistedmatrix.com> Message-ID: <143b1ba2cc8.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> So in conclusion we don't have a particularly solid reason to avoid doing this (might reduce entropy_avail more than normal maybe) and have at least one very good reason (repeated RNG state) to do it? Sent with AquaMail for Android http://www.aqua-mail.com On 20 January 2014 22:09:39 Laurens Van Houtven <_ at lvh.io> wrote: > On Mon, Jan 20, 2014 at 11:04 PM, Glyph Lefkowitz > wrote: > > > FreeBSD and OS X use Yarrow, and Windows uses Fortuna, so nobody but Linux > > even has an exhaustible / blocking random "real actual for real entropy" > > API. You can still exhaust the seed pool but as far as I know it would be > > literally impossible to tell (like, mathematically provably impossible by > > the properties of the various CSPRNGs used) that you had done so. > > > On Linux, /dev/random might still block sometimes, so you can tell that way > > as a regular user, and your applications might get stuck. > > > > At least, as far as I know. The usual caveats about me not being a > > cryptographer, or actually knowing really anything about anything at all, > > apply. > > > > -glyph > > > > > I'm familiar with the CSPRNGs in modern OSes; I just thought we were > talking about exhausting the entropy of the CSPRNGs (so, say, urandom on > Linux and (u)random on FreeBSD and CryptGenRandom on Windows). IIUC, none > of the systems in play here actually use /dev/random in steady state (just > for key generation maybe). If the grandparent indeed meant /dev/random vs > /dev/urandom on Linux that would explain why, yes :-) > > (FYI, I've yet to hear a cryptographer whose opinion on the blocking > /dev/random because better entropy in Linux isn't that it's pretty much > hogwash. djb is convinced they should just give up trying and implement > Fortuna or Yarrow already.) > > cheers > lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at lvh.io Mon Jan 20 23:21:06 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Mon, 20 Jan 2014 23:21:06 +0100 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <143b1ba2cc8.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <143b198a6c0.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> <5159E022-D93B-4ABD-BB0F-00DA7C96DC13@twistedmatrix.com> <143b1ba2cc8.2771.91731227630488f1d17e3e780e9eb50a@prol.etari.at> Message-ID: On Mon, Jan 20, 2014 at 11:16 PM, Alex Stapleton wrote: > So in conclusion we don't have a particularly solid reason to avoid > doing this (might reduce entropy_avail more than normal maybe) and have at > least one very good reason (repeated RNG state) to do it? > Yep. cheers lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-paul at hybridcluster.com Mon Jan 20 23:11:49 2014 From: jean-paul at hybridcluster.com (Jean-Paul Calderone) Date: Mon, 20 Jan 2014 17:11:49 -0500 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <4D8B80093F5E488E88307C148C809B3B@gmail.com> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> Message-ID: <52DD9F25.3050705@hybridcluster.com> On 01/20/2014 03:51 PM, Paul Kehrer wrote: > On Monday, January 20, 2014 at 1:26 PM, Jean-Paul Calderone wrote: >> Since the idea is to use urandom, this won't cause a problem as >> obvious as blocking the SSL server waiting for more entropy. It will >> exhaust what the system (at least on Linux, I don't know much about >> urandom on other platforms) considers the actual amount of available >> entropy. This may mean that any other process that really wants >> /dev/random may be unable to operate. >> Sorry, I think I was unclear here. I don't care about the blocking vs non-blocking nature of urandom and random. I used "/dev/random" to represent the idea of a higher-quality random source (as compared to /dev/urandom). I care about the idea that reading a lot of entropy from either device (again, it doesn't matter which) results in "draining the entropy pool". It sounds like some responders think that "draining the entropy pool" isn't a real thing that can happen and should be ignored. Fine, that's a coherent response. It's somewhat in contrast with (well, directly contradicts) the /dev/urandom man page for Linux but I can accept that the people who wrote that may have been mistaken. :) If that's really what people are saying here? Also, this makes me curious - which device did the people on this list use to generate their private GPG/SSH/SSL/etc keys? /dev/random or /dev/urandom? Jean-Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature URL: From _ at lvh.io Tue Jan 21 00:06:54 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Tue, 21 Jan 2014 00:06:54 +0100 Subject: [Cryptography-dev] OpenSSL Random Engine PR In-Reply-To: <52DD9F25.3050705@hybridcluster.com> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <52DD9F25.3050705@hybridcluster.com> Message-ID: On Mon, Jan 20, 2014 at 11:11 PM, Jean-Paul Calderone < jean-paul at hybridcluster.com> wrote: > I care about the idea that reading a lot of entropy from either device > (again, it doesn't matter which) results in "draining the entropy pool". > It sounds like some responders think that "draining the entropy pool" isn't > a real thing that can happen and should be ignored. Fine, that's a > coherent response. It's somewhat in contrast with (well, directly > contradicts) the /dev/urandom man page for Linux but I can accept that the > people who wrote that may have been mistaken. :) If that's really what > people are saying here? > Draining the entropy pool can happen, but not as a direct consequence of using urandom as suggested here. It's particularly likely to happen in pathological edge cases such as new VMs with no access to a hardware RNG or embedded devices, less likely in the production server scenario outlined earlier in this thread. It can happen, but using a userspace RNG doesn't really immunize you from that problem. Also, this makes me curious - which device did the people on this list use > to generate their private GPG/SSH/SSL/etc keys? /dev/random or > /dev/urandom? > On, OS X or FreeBSD, so, those are the same device for me. There's one more argument for the userspace CSPRNG that I forgot, but it is again unconvincing: you can limit the number of reads you do, and hence syscalls you make. I find it unconvincing because it's a performance argument, and AFAICT no-one is actually having urandom reads be a significant factor anywhere :) hth lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: From glyph at twistedmatrix.com Tue Jan 21 06:41:05 2014 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 20 Jan 2014 21:41:05 -0800 Subject: [Cryptography-dev] Upon the Distinction Between the Entropy of the Various Cryptographically Secure (Or Not) Pseudo (Or Not) Random Number Generators - A Monograph In-Reply-To: <52DD9F25.3050705@hybridcluster.com> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <52DD9F25.3050705@hybridcluster.com> Message-ID: <0D6EA034-D0F9-4491-B9D3-B020FA85718C@twistedmatrix.com> At the risk of being redundant with lvh, I'll try to summarize as well. Mostly I'm doing so because I'd very much like it if the learned participants in this discussion would correct any misconceptions I have; I have been trying to get my head around the _real_ properties of practical portable random number generation for a couple of months now, and I keep heading down blind alleys and understanding things incorrectly. I think I've finally got it here, so this is a good opportunity to find out ;-). Maybe some of these thoughts will even be useful in the documentation. On Jan 20, 2014, at 2:11 PM, Jean-Paul Calderone wrote: > Sorry, I think I was unclear here. I don't care about the blocking vs non-blocking nature of urandom and random. I used "/dev/random" to represent the idea of a higher-quality random source (as compared to /dev/urandom). Assuming that by "blocking vs. non-blocking" what we are actually talking about is "slow vs. fast" (since /dev/random's documentation seems to imply that you can happily set it to O_NONBLOCK and it will give you EAGAIN as normal), the slowness versus the fastness of the source is intimately tied to the level of quality. After all, if there were such a thing as an always-instantly-available high-quality random source, and a slow-so-you-might-have-to-wait high-quality random source, there's no reason I can imagine why you would prefer to use the slow one that might run out, if the other were instantly replenished. My understanding at this point is that we can have such a thing; that a "real" CSPRNG (in the form of the Yarrow and Fortuna algorithms, with appropriate periodic re-seeding, which everyone seems to think is fine on the platforms where they're implemented) gives us such a thing in software, and RDRAND can do it in hardware. On platforms using the yarrow and fortuna algortihms there does not appear to be a "pool" of entropy in the sense we're talking about here; as physical entropy becomes available, it is just used to further re-seed the algorithms immediately. Certainly there's no pool whose exhaustion you can detect from user-land. > I care about the idea that reading a lot of entropy from either device (again, it doesn't matter which) results in "draining the entropy pool". When we were discussing this earlier, I assumed that Linux had implemented a similar CSPRNG, that it was seeded from the hardware entropy sources used to populate /dev/random, and that /dev/random itself must be something of a relic. But, rather than rely on these solutions, it seems that Linux instead implemented some homegrown, weird PRNG for /dev/urandom; and, instead of assuming that the one-way hash functions in this PRNG are, you know, one-way, it still considers "real entropy" to be "used up" by seeding the urandom device and has it removed from the pool so it won't show up on /dev/random, as you suggest; making lots of reads from /dev/urandom can in rare cases cause /dev/random to block. The only paper I could find, from Wikipedia, , suggests that this PRNG has known weaknesses. However, this paper was from 2006, and I know that the algorithm has been changed somewhat recently; I would like to believe that Linux has been modified to be more in line with the security community's consensus assessment of what constitutes a CSPRNG; but I can't find any indication of that. > It sounds like some responders think that "draining the entropy pool" isn't a real thing that can happen and should be ignored. I was hoping that the conclusion of my investigation would be that one can ignore the distinction between /dev/random and /dev/urandom (or equivalent) entirely on all platforms; instead, I appear to have discovered that it's probably still unwise to use /dev/urandom on Linux for any persistent key materials. However, there appears to be consensus that the data from /dev/urandom is suitably random for seeding the CSPRNG in OpenSSL. Going back to the original issue we were originally discussing though, the practical down-side of this particular change to Cryptography is that, on a server (A) being used under heavy load in production, (B) where every request consumes more entropy than it produces, (C) which does not have RDRAND or any other hardware entropy source and (D) running linux, it will be slow to generate persistent key materials because those should be generated from /dev/random and that device may block when its pool is exhausted. This only applies to persistent-key-material generators, because any application using /dev/urandom already needs to be prepared for it to be at this hard-to-qualify slightly-lower-than-random level of randomness anyway, so it isn't a problem if that pool isn't being reseeded as quickly as it should be, and it's apparently "random enough" for almost any other application. The operator of such a system has many possible avenues of attack, without disabling this feature of Cryptography: Stop generating persistent key materials at such a rate on a production server, maybe do that somewhere else, like, once, in advance, seriously, why the heck would you do this Switch to FreeBSD Switch to OS X Buy an external hardware random number generator Upgrade to hardware that has RDRAND Given the peculiar obscurity of this use-case and also the multiple avenues open to the operator, it seems reasonable to me to proceed with the change. > Fine, that's a coherent response. It's somewhat in contrast with (well, directly contradicts) the /dev/urandom man page for Linux but I can accept that the people who wrote that may have been mistaken. :) If that's really what people are saying here? I am trusting them to make accurate statements about their own implementation - entropy pool depletion is in fact a real thing that happens sometimes on Linux. However, it appears to be something of a bug, and I'm a bit confused as why they are intent on keeping this feature when it has pretty much disappeared or never existed on other platforms, for many, many years. Plus, perhaps more importantly, entropy pool deletion is purely a performance concern. Correctly-written code will select the appropriate source of randomness, and /dev/urandom is apparently good enough for almost everything but long-lived secret key generation. Which means that the only thing that might happen if you deplete the pool is that something demanding a high degree of randomness might have to wait for a little while; and even this performance concern is only an issue in configurations where there is no further readily-available source of entropy to replenish the pool; such sources can be reasonably easily acquired if an application requires it. > Also, this makes me curious - which device did the people on this list use to generate their private GPG/SSH/SSL/etc keys? /dev/random or /dev/urandom? After this little survey of the literature, I'm going to declare this question to be poorly phrased and say that I generated it with securityd and Yarrow, since apparently the name of the device is meaningless ;-). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at lvh.io Tue Jan 21 11:54:48 2014 From: _ at lvh.io (Laurens Van Houtven) Date: Tue, 21 Jan 2014 11:54:48 +0100 Subject: [Cryptography-dev] Upon the Distinction Between the Entropy of the Various Cryptographically Secure (Or Not) Pseudo (Or Not) Random Number Generators - A Monograph In-Reply-To: <0D6EA034-D0F9-4491-B9D3-B020FA85718C@twistedmatrix.com> References: <52DD786F.5060802@hybridcluster.com> <4D8B80093F5E488E88307C148C809B3B@gmail.com> <52DD9F25.3050705@hybridcluster.com> <0D6EA034-D0F9-4491-B9D3-B020FA85718C@twistedmatrix.com> Message-ID: On Tue, Jan 21, 2014 at 6:41 AM, Glyph wrote: > My understanding at this point is that we *can* have such a thing; that a > "real" CSPRNG (in the form of the Yarrow and Fortuna algorithms, with > appropriate periodic re-seeding, which everyone seems to think is fine on > the platforms where they're implemented) gives us such a thing in software, > and RDRAND can do it in hardware. > Yes. > On platforms using the yarrow and fortuna algortihms there does not appear > to be a "pool" of entropy in the sense we're talking about here; as > physical entropy becomes available, it is just used to further re-seed the > algorithms immediately. > That may not be entirely true, depending on what exactly you mean with "not have a pool of entropy in the sense we're talking about here". Yarrow has two pools in which entropy accumulates. The pools produce keys, which are fed to a PRNG to produce the output random numbers. Reseeds happen periodically, not immediately. (See sections 4.2, 4.4 of the Yarrow paper.) It's closer to the truth in Fortuna, which has many (32 by default) smaller pools. Fortuna does *not* attempt to estimate how much entropy there is; instead, it is designed so that if there's enough entropy in *some* of the inputs, it would still be secure. Fortuna by default also places limits on reseeding, but otherwise entropy collection can lead directly to reseeding. I only gave the rest of the e-mail a casual once-over, but didn't see anything that rustled my jimmies. cheers lvh -------------- next part -------------- An HTML attachment was scrubbed... URL: