[Cryptography-dev] Thread safety in the OpenSSL backend

[Alex Stapleton]

AIUI the cffi bindings release the GIL during the C call. This allows 
another thread to run that can then end up calling into OpenSSL again. If 
we leave the GIL locked during the call there's no way for there to be 
concurrent calls. Gives you 0 parallelism on all operations though.

Sent with AquaMail for Android
http://www.aqua-mail.com


On 20 January 2014 18:26:13 Alex Gaynor <alex.gaynor at gmail.com> wrote:
> Maybe a method on `Binding` is the right place? Not sure, let's get the
> code up, then we can find a place.
>
> What do you mean "Enable the GIL"?
>
> Alex
>
>
> On Mon, Jan 20, 2014 at 12:25 PM, Alex Stapleton <alexs at prol.etari.at>wrote:
>
> >  I've already implemented 2 as well so its not too awful :-)  Needs tests
> > and things though.
> >
> > The current PR is all exposed on the binding so PyOpenSSL could easily use
> > it. Where do you suggest a Python implementation should live? A utils
> > module in the binding.openssl package?
> >
> > I failed to mention option 4 in my previous post too.
> >
> > 4. Enable the GIL on all of the OpenSSL bindings.
> >
> > Not entirely sure how you convince cffi to do that though. Might not be
> > possible?
> >
> > Sent with AquaMail for Android
> > http://www.aqua-mail.com
> >
> > On 20 January 2014 16:10:44 Alex Gaynor  wrote:
> >
> > How much would you hate me if I asked about doing a prototype of #2? It's
> > the kind of performance issue that we can fix later very easily, but makes
> > it much easier to review now.
> >
> > In general +1 on us fixing this, one suggestion though: this shouldn't be
> > done in backend, it should be done somewhere ~~generic~~ so that pyOpenSSL
> > can just call this function without needing to instantiating a backend.
> >
> > Alex
> >
> >
> > On Mon, Jan 20, 2014 at 10:07 AM, alexs <alexs at prol.etari.at> wrote:
> >
> >> TLDR; We would like to merge a thread safety solution for the OpenSSL
> >> bindings [1].
> >>
> >> Probably this https://github.com/pyca/cryptography/pull/377
> >>
> >> What do we need to do to get it reviewed and in master?
> >>
> >> As has been recently discovered by Hynek [2], initialising OpenSSL is
> >> more difficult than it should be. This is complicated by the Python VM
> >> doing its own initialisation of OpenSSL things inside the _ssl module.
> >> Normally this works in our favour as Python happens to call the same
> >> functions as us. If we have linked cryptography to a different version of
> >> OpenSSL to the one Python was compiled against (such as with bottled
> >> OpenSSL) however this can result in _ssl not doing any initialisation for
> >> us. In particular, it wont setup the threading callback required to make
> >> (among other things) the OpenSSL RNG thread safe. This is particularly
> >> annoying on OS X which ships with OpenSSL 0.9.8 still.
> >>
> >> There are a few approaches to this.
> >>
> >>  1. Ignore it, only support Python OpenSSL, document that somewhere. Will
> >> probably make OS X users sad.
> >>  2. Expose the callback setup functions and implement locking entirely in
> >> Python. This has 4-8x more overhead than the C version but is a small
> >> amount of code, relatively easy to test and works cross-platform.
> >>  3. Implement our own cross-platform locking wrappers and point OpenSSL
> >> at those. This is fast but is harder to test well while we have no Windows
> >> build infrastructure. Also puts even more code in C which we have no
> >> coverage data on.
> >>
> >> I have written a PR that implements option 3: https://github.com/pyca/
> >> cryptography/pull/377
> >>
> >> The code was mostly borrowed from PyPy, which is mostly borrowed from an
> >> old version of CPython. It seems to solve my small crashing test cases on
> >> POSIX.
> >>
> >> We could really use some people who feel confident (or perhaps even
> >> competent) with threading/C/POSIX/WINAPI/OpenSSL etc to review this.
> >> It's not had a lot of eyes on and I'm particularly uncomfortable with the
> >> Windows parts since I lack the hardware (or software licenses) to test them.
> >>
> >> [1] https://github.com/pyca/cryptography/issues/330
> >> [2] https://github.com/pyca/pyopenssl/issues/7
> >> _______________________________________________
> >> Cryptography-dev mailing list
> >> Cryptography-dev at python.org
> >> https://mail.python.org/mailman/listinfo/cryptography-dev
> >>
> >
> >
> >
> > --
> > "I disapprove of what you say, but I will defend to the death your right
> > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> > "The people's good is the highest law." -- Cicero
> > GPG Key fingerprint: 125F 5C67 DFE9 4084
> >
> >
> > _______________________________________________
> > Cryptography-dev mailing list
> > Cryptography-dev at python.org
> > https://mail.python.org/mailman/listinfo/cryptography-dev
> >
> >
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20140120/442b3b93/attachment.html>