[Cryptography-dev] Bundling OpenSSL

[Donald Stufft]

On Oct 20, 2013, at 9:44 AM, Jean-Paul Calderone <jean-paul at hybridcluster.com> wrote:

> On 10/19/2013 08:26 PM, Donald Stufft wrote:
>> 
>> I've started working on bundling OpenSSL within cryptography and fixing the packaging up.
>> 
>> There are a number of things I'd like to do that will make things smoother in the long run but require some changes right now. These things are:
>> 
>> Disable implicit compile
>>     - This masks problems with the packaging. I tried leaving it on when doing the same thing for PyNaCl and it caused immense frustration with strange edge cases breaking. Further more the cffi devs say they are planning on removing it anyways.
> I agree that implicit compile is a little bit crazy.  It ends up invoking a ton of code when the user probably doesn't expect it.
> 
> The only concern I have with disabling this is that it sounds like the outcome of doing it naively means that source and compiled modules can easily get out of sync resulting in confusing errors.
> 
> It would be nice if there could at *least* be a warning emitted if the source and compiled modules are mismatched.

I'm not sure how to do this warning. Do you know of a way? But yes it does mean that things can get out of sync.

> 
>> Move the cryptography directory into a src directory
>>     - Since cryptography will require an implicit compile step with the above, it's easy to accidentally import the cryptography that is sitting on the file system that hasn't been compiled yet, moving it means you have to install cryptography somehow to import it (or cd into the src directory). This should lower confusing errors when ``python -c 'import cryptography'`` does work. Further since we'll need a folder for the OpenSSL code it makes sense I think to have a single folder for all the things we have in repo, see: https://github.com/pyca/pynacl/tree/master/src
> 
> I don't like this idea.  It sounds like the reasoning behind it is "explicitly compilation makes things more inconvenient so we should re-arrange the repository to make things even *more* inconvenient than that" and then somehow conflating "inconvenient" with "not confusing".  Moreover, I don't find the explanation that people will be confused if they try to use cryptography without compiling it compelling.  If you want to remove confusing errors from that usage, detect missing compiled artifacts and raise an explicit exception about the problem.
> 
> "src" directories are a holdover from ancient history when tools worked differently.  It's almost always pointless now (even if the entire project is written in C).

I'm neutral on src directories. The primary reason I want to do it here is from my experience in fixing PyNaCl is it got confusing for me personally. If I ran ``pip install .`` and then tried to import nacl I would get an error because Python automatically added the current directory to the path which meant nacl tried to import the copy sitting on my disk which didn't have a compiled library.

This isn't a requirement for sane packaging or bundling OpenSSL though, I just think it's cleaner to get rid of that potential pain point as well as provide a single directory for all of the source things we bundle.

> 
>> 
>> Lazily load the compiled library
>>     - Right now the compiled library is loaded when the cryptography.bindings.openssl.api library is imported, this is the way CFFI currently works. As i've done in PyNaCl i'd like to make the Api object lazy so that it loads the compiled library on first access and proxies the calls to the exposed functions/variables to the lazily loaded underlying library object. You can see this at: https://github.com/pyca/pynacl/blob/master/src/nacl/_lib/__init__.py#L59-L84
> 
> Why?  If someone imports cryptography.bindings.openssl.api then isn't it pretty likely they're going to use something from the compiled library?  What is the scenario in which this results in less work being done?

Essentially the way cffi works at the moment is you have to import the cryptography.bindings.openssl.api to get the distutils Extension class. If implicit compile is disabled then if importing that module attempts to load the compiled library (which it will how the code is currently) which will fail because it doesn't exist. Essentially we need to either lazily load, duplicate code into the setup.py, leave implicit compile on, or move the creation of the library instance to another module. (FFI verify implicit creates the library, but that can be moved).

> 
> Jean-Paul
> 
>> The final solution to this would be that we have a library that is always built at build time with a consistent version of OpenSSL. 
>> 
>> Thoughts on these steps? 
>> 
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>> 


An additional question is wether we want to bundle the OpenSSL source unpacked in the source tree (like I did with libsodium in PyNacl) or if we want to just include a tarball and unpack it at build time to a temporary directory?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20131020/c27b7353/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20131020/c27b7353/attachment-0001.sig>