From djve60 at gmail.com Sat Jan 3 00:13:13 2015 From: djve60 at gmail.com (David Evans) Date: Fri, 2 Jan 2015 15:13:13 -0800 Subject: [Cryptography-dev] Fernet newbie question: suppling existing key for decoding Message-ID: I'm playing with the Fernet and AES encryption from cryptography. I've only started with Python a few weeks ago and I'm not up on cryptography. I'm trying to decrypt data that has been encoded with OpenSSL. I can decode it with python 2.7 using Crypto.Cipher but I'm having problems getting cryptography to do the same thing. Q: While key *=* Fernet*.*generate_key() will generate a new key is there a way to generate a valid key from an existing passphrase? openssl enc -base64 produces a key that is too small. Does this need to be padded with NULLs or is there a command I'm missing? I'd prefer documentation that will bring me up to speed on this area rather than example code. The November thread on signatures and keys didn't address my simplistic issue. FYI: I have tried using the lower level "Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend)" but my decryptor generates and error but this due to my sending a str and not bytes. Many thanks, David -- ========================= Your reason is treason. Kasabian -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.l.kehrer at gmail.com Sat Jan 3 03:50:24 2015 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Fri, 2 Jan 2015 20:50:24 -0600 Subject: [Cryptography-dev] Fernet newbie question: suppling existing key for decoding In-Reply-To: References: Message-ID: Hi David, Fernet is a recipe that implements the Fernet specification (https://github.com/fernet/spec/blob/master/Spec.md). Accordingly you will only be able to use it to decrypt things that have been encrypted with other libraries that utilize that specification. At this time there's no (official) way to derive a Fernet key from a passphrase (although you could look at how Fernet keys are generated and use the KDF primitives to accomplish such a thing). If you supply your PyCrypto code we can more clearly understand what you're trying to accomplish, but in the absence of that here are some avenues of inquiry to pursue... openssl enc derives keys and IVs from passphrases in its own special way (see:?http://security.stackexchange.com/questions/29106/openssl-recover-key-and-iv-by-passphrase). You are correct in thinking you need to use the hazmat APIs to do this decryption (assuming you have a hard requirement of being able to decrypt data encrypted using the openssl enc utility). To accomplish that you'll need to write some code that can replicate the EVP_BytesToKey (https://www.openssl.org/docs/crypto/EVP_BytesToKey.html) call that enc uses (https://github.com/openssl/openssl/blob/master/apps/enc.c#L569). The implementation of that function can be found here (https://github.com/openssl/openssl/blob/e0fc7961c4fbd27577fb519d9aea2dc788742715/crypto/evp/evp_key.c#L115). If the user leaves everything default this function is the equivalent of MD5(password+salt) If, on the other hand, you just generically want to be able to encrypt data and have it be decrypted by libraries in other languages you can either use Fernet (and decrypt it with something like?https://github.com/fernet/fernet-rb,?https://github.com/fernet/fernet-go, or?https://github.com/dreid/fernet-clj) or generate a key/iv yourself (using os.urandom) and then use them to encrypt. Looking through our docs we appear to be missing an example of this using our padding functions with our encryptor/decryptor so here you go: padder = padding.PKCS7(algorithms.AES.block_size).padder() cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend) encryptor = cipher.encryptor() # this next line is all in one, but if you're streaming large volumes of data you'd split # it up to make as many update calls as needed ct = encryptor.update(padder.update(data) + padder.finalize()) + encryptor.finalize() To decrypt you'll need the ct, IV, and key. Of course, this isn't authenticated encryption (which you get with Fernet or an AEAD mode like GCM) so you'll either need to add an HMAC to this or switch to GCM if you want to have confidence in the message integrity. (This is why constructs like Fernet are valuable and why we recommend them over rolling your own if possible). -Paul Kehrer On January 2, 2015 at 8:00:45 PM, David Evans (djve60 at gmail.com) wrote: I'm playing with the Fernet and AES encryption from cryptography. I've only started with Python a few weeks ago and I'm not up on cryptography. I'm trying to decrypt data that has been encoded with OpenSSL. I can decode it with python 2.7 using Crypto.Cipher but I'm having problems getting cryptography to do the same thing. Q: While ? ?key = Fernet.generate_key() will generate a new key is there a way to generate a valid key from an existing passphrase? ? ?openssl enc -base64? produces a key that is too small. Does this need to be padded with NULLs or is there a command I'm missing? I'd prefer documentation that will bring me up to speed on this area rather than example code. The November thread on signatures and keys didn't address my simplistic issue. FYI: I have tried using the lower level "Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend)" but my decryptor generates and error but this due to my sending a str and not bytes. Many thanks, David -- ========================= Your reason is treason. Kasabian _______________________________________________ Cryptography-dev mailing list Cryptography-dev at python.org https://mail.python.org/mailman/listinfo/cryptography-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From djve60 at gmail.com Mon Jan 5 19:05:40 2015 From: djve60 at gmail.com (David Evans) Date: Mon, 5 Jan 2015 10:05:40 -0800 Subject: [Cryptography-dev] Fernet newbie question: suppling existing key for decoding In-Reply-To: References: Message-ID: I found the answer to what I was trying to find at ? http://incolumitas.com/2014/10/19/using-the-python-cryptography-module-with-custom-passwords/ when following up on the URLs. In case the site goes off line the code I was after is just (copied from the site) by Nikolai Tschacher: import base64 from cryptography.fernet import Fernet from cryptography.hazmat.primitives import hashes from cryptography.hazmat.backends import default_backend def get_key(password): digest = hashes.Hash(hashes.SHA256(), backend=default_backend()) digest.update(password) return base64.urlsafe_b64encode(digest.finalize()) def encrypt(password, token): f = Fernet(get_key(key)) return f.encrypt(bytes(token)) def decrypt(password, token): f = Fernet(get_key(password)) return f.decrypt(bytes(token)) but the problem I have with this is using a non-randomized password helps subvert the security of the 32-bit random password that Fernet provides. However, as Nikolai notes, it's more usable for interactive activities. For new code I'll be using Fernet but for existing code I think I'm forced to keep to PyCrypto. Since a Fernet message is a self-signed (a little like a PKCS12 type file) it will not be appropriate for many existing processes but I think it should be used where possible. Thanks for the help, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.l.kehrer at gmail.com Mon Jan 5 19:26:20 2015 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Mon, 5 Jan 2015 12:26:20 -0600 Subject: [Cryptography-dev] Fernet newbie question: suppling existing key for decoding In-Reply-To: References: Message-ID: Thanks for the update David. I'd strongly recommend against using that construction as it is just doing a single SHA256 hash of your password and then base64 encoding it. While a password is definitely lower entropy than 32 bytes of random you can still derive a key suitable for encryption from a password by using a KDF (key derivation function). KDFs are useful in this context because they raise the cost of computing the key so that an attacker can't easily brute force the password. Consider replacing the get_key function with something using PBKDF2HMAC (https://cryptography.io/en/latest/hazmat/primitives/key-derivation-functions/#cryptography.hazmat.primitives.kdf.pbkdf2.PBKDF2HMAC). An even better KDF would be something like scrypt, but unfortunately we don't support that in pyca/cryptography yet. This definitely is showing a need for an official password-based encryption key derivation for Fernet... -Paul Kehrer On January 5, 2015 at 12:07:31 PM, David Evans (djve60 at gmail.com) wrote: I found the answer to what I was trying to find at??http://incolumitas.com/2014/10/19/using-the-python-cryptography-module-with-custom-passwords/ when following up on the URLs. In case the site goes off line the code I was after is just (copied from the site) by Nikolai Tschacher: import base64 from cryptography.fernet import Fernet from cryptography.hazmat.primitives import hashes from cryptography.hazmat.backends import default_backend def get_key(password): ? ? digest = hashes.Hash(hashes.SHA256(), backend=default_backend()) ? ? digest.update(password) ? ? return base64.urlsafe_b64encode(digest.finalize()) def encrypt(password, token): ? ? f = Fernet(get_key(key)) ? ? return f.encrypt(bytes(token)) def decrypt(password, token): ? ? f = Fernet(get_key(password)) ? ? return f.decrypt(bytes(token)) but the problem I have with this is using a non-randomized password helps subvert the security of the 32-bit random password that Fernet provides. However, as Nikolai notes, it's more usable for interactive activities. For new code I'll be using Fernet but for existing code I think I'm forced to keep to PyCrypto. Since a Fernet message is a self-signed (a little like a PKCS12 type file) it will not be appropriate for many existing processes but I think it should be used where possible.? Thanks for the help, David _______________________________________________ Cryptography-dev mailing list Cryptography-dev at python.org https://mail.python.org/mailman/listinfo/cryptography-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.l.kehrer at gmail.com Fri Jan 16 15:24:25 2015 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Fri, 16 Jan 2015 08:24:25 -0600 Subject: [Cryptography-dev] PyCA cryptography 0.7.2 release Message-ID: PyCA cryptography 0.7.2 has been released to PyPI. Changelog: * Updated Windows wheels to be compiled against OpenSSL 1.0.1l. * enum34 is no longer installed on Python 3.4, where it is included in the standard library. * Added a new function to the OpenSSL bindings to support additional functionality in pyOpenSSL. As always, thanks to everyone who helped with this release. -Paul Kehrer -------------- next part -------------- An HTML attachment was scrubbed... URL: From dr.michael.iverson at gmail.com Mon Jan 19 16:19:08 2015 From: dr.michael.iverson at gmail.com (Michael Iverson) Date: Mon, 19 Jan 2015 10:19:08 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. Message-ID: Hello, I'm new to the cryptography library, and I'm definitely excited about having a well-reviewed cryptographic library with a simple API. However, I'm noticing that there is area that might improve the usefulness of the high level methods. The Fernet API is where my interest lies, as I'm presently in need of a symmetric algorithm. However, the idea could be equally applicable for other APIs. The fernet API presently requires that a complete buffer containing the plaintext or cyphertext be passed to the encryption or decryption methods. This requirement becomes memory inefficient for moderately sized objects, and can prevent processing of large objects entirely, especially on memory constrained systems. Furthermore, many python libraries use file handles as an abstraction for incrementally consuming or producing data. Examples include http responses in Tornado and Cyclone, and the SFTP interface in paramiko. I'd like to propose the addition of an alternate API that would accept and return file handles, and incrementally encrypt or decrypt using the handles. I think this would make the library more useful for a variety of solutions, enhancing adoption. On the surface, it appears that the main cryptographic primitives, (hmac, padding, aes, etc.) are designed to operate on in an incremental fashion, using the update() method to incrementally compute data, and finalize() to return the final result, so the change may not be overly difficult. I'm willing to contribute the code for such an endeavor, as I'm going to write it anyway for a current project. Contributing the code will help ensure it it is adequately reviewed. Does anyone feel this would be a worthwhile improvement? -- Michael Iverson -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-paul at clusterhq.com Mon Jan 19 16:47:13 2015 From: jean-paul at clusterhq.com (Jean-Paul Calderone) Date: Mon, 19 Jan 2015 10:47:13 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: References: Message-ID: <87lhkypybi.fsf@phonon.i-did-not-set--mail-host-address--so-tickle-me> Michael Iverson writes: > Hello, > > [snip] > > Furthermore, many python libraries use file handles as an abstraction for > incrementally consuming or producing data. Examples include http responses > in Tornado and Cyclone, and the SFTP interface in paramiko. > > I'd like to propose the addition of an alternate API that would accept and > return file handles, and incrementally encrypt or decrypt using the > handles. I think this would make the library more useful for a variety of > solutions, enhancing adoption. > A file-like interface is not particularly well-suited for asynchronous operation. Have you looked into whether Tornado and Cyclone would actually be able to use such an interface? This is not to say such an interface shouldn't be introduced - but if two out of three of the potentially users can't actually use it, it might not be as widely applicable as you guessed. And it would certainly be bad if this were the *only* interface Cryptography provided to certain functionality (which, fortunately, isn't what you're suggesting, I think). Jean-Paul From donald at stufft.io Mon Jan 19 16:51:19 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 Jan 2015 10:51:19 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: References: Message-ID: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> > On Jan 19, 2015, at 10:19 AM, Michael Iverson wrote: > > Hello, > > I'm new to the cryptography library, and I'm definitely excited about having a well-reviewed cryptographic library with a simple API. > > However, I'm noticing that there is area that might improve the usefulness of the high level methods. The Fernet API is where my interest lies, as I'm presently in need of a symmetric algorithm. However, the idea could be equally applicable for other APIs. > > The fernet API presently requires that a complete buffer containing the plaintext or cyphertext be passed to the encryption or decryption methods. This requirement becomes memory inefficient for moderately sized objects, and can prevent processing of large objects entirely, especially on memory constrained systems. > > Furthermore, many python libraries use file handles as an abstraction for incrementally consuming or producing data. Examples include http responses in Tornado and Cyclone, and the SFTP interface in paramiko. > > I'd like to propose the addition of an alternate API that would accept and return file handles, and incrementally encrypt or decrypt using the handles. I think this would make the library more useful for a variety of solutions, enhancing adoption. > > On the surface, it appears that the main cryptographic primitives, (hmac, padding, aes, etc.) are designed to operate on in an incremental fashion, using the update() method to incrementally compute data, and finalize() to return the final result, so the change may not be overly difficult. > > I'm willing to contribute the code for such an endeavor, as I'm going to write it anyway for a current project. Contributing the code will help ensure it it is adequately reviewed. > > Does anyone feel this would be a worthwhile improvement? The problem with streaming APIs (and why the recipes layer doesn?t currently have anything to work with them) is that it gives data to the user prior to it being authenticated. For example, let?s say someone modifies the first chunk of a big file that is encrypted, and you do something like: with open(?decrypted.txt?, ?wb?) as d_fp: with open(?encrypted?, ?rb?) as s_fp: for chunk in cryptography.decrypt_file(s_fp): s_fp.write(chunk) This is a fairly obvious way of handling that. However it?ll write a whole bunch of data to decrypted.txt and only fail after the very last chunk. Software or humans might not notice that and start operating on that data even though it?s been authenticated (perhaps instead of writing to a file it?s writing to stdout and you?re using it in a unix pipe or something). The attractive-ness of the current API for fernet is that you either get back completely authenticated data or you don?t so there is no ?oops an error happened but I didn?t notice and operated on data that was invalid?. As you noticed, this starts to break down for objects that are large, especially ones that are so large they cannot be reasonable held completely in memory. Currently, due to the above issues, handling large objects requires using the hazmat API because we haven?t yet come up with a ?safe? way of handling large objects and have it implemented. One likely candidate is to do something similar to what TLS does, and that is break the large objects up into chunks of a certain size, and basically encrypt and authenticate each chunk separately. Since each chunk operates independently, you essentially are calling the ?one shot? APIs for each chunk, and only hand back each chunk to the caller of the API when it?s been fully authenticated. This allows us to control how much we need to buffer in memory at any one time (basically the size of the chunk).This would also need some higher level authentication across all of the chunks or a serial number or something that will ensure that chunks don?t arrive out or order or missing a particular chunk. Ideally we?d use something like the above scheme that is backed by some sort of standard, whether an RFC or some thing that doesn?t involve us creating our own format. However it might be the case that we need to create our own format, and if we do then we should attempt to make that a standard and get it reviewed by cryptographers to make sure that we aren?t forgetting something. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From dr.michael.iverson at gmail.com Mon Jan 19 17:16:39 2015 From: dr.michael.iverson at gmail.com (Michael Iverson) Date: Mon, 19 Jan 2015 11:16:39 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> References: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> Message-ID: On Mon, Jan 19, 2015 at 10:51 AM, Donald Stufft wrote: > This is a fairly obvious way of handling that. However it?ll write a whole > bunch of data to decrypted.txt and only fail after the very last chunk. That is definitely a concern, and it cannot be readily mitigated, as not keeping everything in memory is exactly what is required. However, I'm not sure the chunk based approach necessarily mitigates this problem either, as you could write out hundreds of chunks, only to have the final chunk fail. Also, having multiple chunks also requires that we somehow manage to ensure that we can identify missing or out-of-order chunks. I'd also be concerned about the cryptographic implications of this. I'm not sure if this is entirely correct, but it seems if you set your chunk size = AES block size, you essentially are encrypting in ECB mode. I would presume there is a block size sufficiently large to mitigate this problem, but I get chills up my spine when I use the word 'presume' in any sentence about cryptography. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jarret.raim at RACKSPACE.COM Mon Jan 19 17:25:45 2015 From: jarret.raim at RACKSPACE.COM (Jarret Raim) Date: Mon, 19 Jan 2015 16:25:45 +0000 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: References: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> Message-ID: Donald's point was that each chunk is authenticated separately. As soon as a single chunk failed, the decryption operation would fail ? you wouldn't have to wait until the end of the file. From: Michael Iverson Reply-To: "cryptography-dev at python.org" Date: Monday, January 19, 2015 at 10:16 AM To: "cryptography-dev at python.org" Subject: Re: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. On Mon, Jan 19, 2015 at 10:51 AM, Donald Stufft > wrote: This is a fairly obvious way of handling that. However it?ll write a whole bunch of data to decrypted.txt and only fail after the very last chunk. That is definitely a concern, and it cannot be readily mitigated, as not keeping everything in memory is exactly what is required. However, I'm not sure the chunk based approach necessarily mitigates this problem either, as you could write out hundreds of chunks, only to have the final chunk fail. Also, having multiple chunks also requires that we somehow manage to ensure that we can identify missing or out-of-order chunks. I'd also be concerned about the cryptographic implications of this. I'm not sure if this is entirely correct, but it seems if you set your chunk size = AES block size, you essentially are encrypting in ECB mode. I would presume there is a block size sufficiently large to mitigate this problem, but I get chills up my spine when I use the word 'presume' in any sentence about cryptography. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Jan 19 17:27:45 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 Jan 2015 11:27:45 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: References: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> Message-ID: <9E690175-57CE-4C70-A441-7A7F287E4861@stufft.io> > On Jan 19, 2015, at 11:16 AM, Michael Iverson wrote: > > > On Mon, Jan 19, 2015 at 10:51 AM, Donald Stufft > wrote: > This is a fairly obvious way of handling that. However it?ll write a whole bunch of data to decrypted.txt and only fail after the very last chunk. > > That is definitely a concern, and it cannot be readily mitigated, as not keeping everything in memory is exactly what is required. > > However, I'm not sure the chunk based approach necessarily mitigates this problem either, as you could write out hundreds of chunks, only to have the final chunk fail. Also, having multiple chunks also requires that we somehow manage to ensure that we can identify missing or out-of-order chunks. The point here is while we might fail on the very last chunk, we ensure that each chunk is fully authenticated before we give it to the user. So yes, the user can operate on decrypted and authenticated data which might ultimately fail on the last chunk, however they will not be able to operate on decrypted and unauthenticated data which might ultimately fail on the last chunk. There is no way that I am aware of to create an API that allows streaming but also prevents the user from reading the streamed data prior to having processed the entire stream. The best that I can think of is one that prevents the user from reading data that we haven?t authenticated to make sure it wasn?t modified in transit. > > I'd also be concerned about the cryptographic implications of this. I'm not sure if this is entirely correct, but it seems if you set your chunk size = AES block size, you essentially are encrypting in ECB mode. The actual details of what you?d need to do is more involved than just calling encrypt() with the same key on chunks. That?s just a high level ?here?s the general idea thing?. In reality you?d encrypt the stream using the streaming encryption APIs (so you?d use something like CBC or CTR) and you?d take that output and break it into chunks as well, and you?d authenticate each of those chunks. In pseudo code it?s look something like: def safe_streaming_encrypt(plaintext_iterator): for encrypted_chunk in unauthenticated_streaming_encrypt(plaintext_iterator): yield HMAC(encypted_chunk), encrypted_chunk You?d need this to actually yield bytes which represents a ?record? (e.g. an encoded representation of the data of the encrypted chunk, the authentication tag, and anything else the scheme needs). However this would ensure that you?d never actually give the user unauthenticated data. > > I would presume there is a block size sufficiently large to mitigate this problem, but I get chills up my spine when I use the word 'presume' in any sentence about cryptography. > --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From dr.michael.iverson at gmail.com Mon Jan 19 17:30:44 2015 From: dr.michael.iverson at gmail.com (Michael Iverson) Date: Mon, 19 Jan 2015 11:30:44 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: <87lhkypybi.fsf@phonon.i-did-not-set--mail-host-address--so-tickle-me> References: <87lhkypybi.fsf@phonon.i-did-not-set--mail-host-address--so-tickle-me> Message-ID: On Mon, Jan 19, 2015 at 10:47 AM, Jean-Paul Calderone < jean-paul at clusterhq.com> wrote: > > A file-like interface is not particularly well-suited for asynchronous > operation. Have you looked into whether Tornado and Cyclone would > actually be able to use such an interface? > > I don't see an immediate reason why it could not. I could be wrong on this point, however. For example, here is an excerpt from a simple twisted protocol I wrote. The protocol spawns an external process, and reads data from stdout. The class has a attribute 'request', which is the file handle generating the HTTP response. For each chunk of data it reads, it (asynchronously) calls def outReceived(self, data): self.request.write(data) self.request.flush() Most of the asynchronous behavior involves waiting for IO or network responses, and processing chunks (or entire responses) when available. This seems to fit this proposal well. > And it would certainly be bad if this were the *only* interface > Cryptography provided to certain functionality (which, fortunately, > isn't what you're suggesting, I think). > That is definitely not what I'm suggesting. Just an alternate interface for a specific option. > > Jean-Paul > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev > -- Dr. Michael Iverson Director of Information Technology Hatteras Printing -------------- next part -------------- An HTML attachment was scrubbed... URL: From dr.michael.iverson at gmail.com Mon Jan 19 17:42:41 2015 From: dr.michael.iverson at gmail.com (Michael Iverson) Date: Mon, 19 Jan 2015 11:42:41 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: <9E690175-57CE-4C70-A441-7A7F287E4861@stufft.io> References: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> <9E690175-57CE-4C70-A441-7A7F287E4861@stufft.io> Message-ID: On Mon, Jan 19, 2015 at 11:27 AM, Donald Stufft wrote: > > I'd also be concerned about the cryptographic implications of this. I'm > not sure if this is entirely correct, but it seems if you set your chunk > size = AES block size, you essentially are encrypting in ECB mode. > > > The actual details of what you?d need to do is more involved than just > calling encrypt() with the same key on chunks. That?s just a high level > ?here?s the general idea thing?. In reality you?d encrypt the stream using > the streaming encryption APIs (so you?d use something like CBC or CTR) and > you?d take that output and break it into chunks as well, and you?d > authenticate each of those chunks. > > > OK, I see what you're saying now. You would continue to operate the cipher in CBC mode, and process chunks in order. You would then HMAC each chunk, so that each could be validated prior to decryption and writing that chunk to the output. No unvalidated data would ever be passed downstream. At worst, you would have partial data. To handle missing or out of order chunks, you could maintain an hmac digest of all blocks processed prior to the current block, in addition to the added block. A missing block would cause the HMAC to fail. There would also need to be a method to ensure that we only a certain number of chunks. -- Dr. Michael Iverson Director of Information Technology Hatteras Printing -------------- next part -------------- An HTML attachment was scrubbed... URL: From dr.michael.iverson at gmail.com Mon Jan 19 17:48:10 2015 From: dr.michael.iverson at gmail.com (Michael Iverson) Date: Mon, 19 Jan 2015 11:48:10 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: References: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> <9E690175-57CE-4C70-A441-7A7F287E4861@stufft.io> Message-ID: > > > To handle missing or out of order chunks, you could maintain an hmac > digest of all blocks processed prior to the current block, in addition to > the added block. A missing block would cause the HMAC to fail. There would > also need to be a method to ensure that we only a certain number of chunks. > > Sorry, the last sentence was incoherent. I meant to say you'd need to be able to identify a truncated stream of chunks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jean-paul at clusterhq.com Mon Jan 19 18:09:52 2015 From: jean-paul at clusterhq.com (Jean-Paul Calderone) Date: Mon, 19 Jan 2015 12:09:52 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: References: <87lhkypybi.fsf@phonon.i-did-not-set--mail-host-address--so-tickle-me> Message-ID: <87ppaay9wf.fsf@phonon.i-did-not-set--mail-host-address--so-tickle-me> Michael Iverson writes: > On Mon, Jan 19, 2015 at 10:47 AM, Jean-Paul Calderone < > jean-paul at clusterhq.com> wrote: > >> >> A file-like interface is not particularly well-suited for asynchronous >> operation. Have you looked into whether Tornado and Cyclone would >> actually be able to use such an interface? >> >> > I don't see an immediate reason why it could not. I could be wrong on this > point, however. > > For example, here is an excerpt from a simple twisted protocol I wrote. The > protocol spawns an external process, and reads data from stdout. > The class has a attribute 'request', which is the file handle generating > the HTTP response. > > For each chunk of data it reads, it (asynchronously) calls > > def outReceived(self, data): > self.request.write(data) > self.request.flush() > > Most of the asynchronous behavior involves waiting for IO or network > responses, and processing chunks (or entire responses) when available. This > seems to fit this proposal well. > Hm. I'm not sure I quite follow how this is applicable. It may be that I've misunderstood the interface you're suggesting. In this example, the source of input is asynchronous (`outReceived` events) and the destination for output is non-blocking (`request.write` and `request.flush`). On both sides, there are no considerations for flow-control (though I happen to know that `self` can probably do something like `self.transport.pauseProducing()` if it wants to slow down the input source). Without flow control, you may end up with your entire dataset buffered in memory anyway. The traditional way flow control is implemented with the file interface is by making `read` or `write` into a blocking operation. So, I think flow control is one general shortcoming of the file-based interface. A related concern is the tendancy to write asynchronous unfriendly code using file-based interfaces. If someone writes a handy utility that wants to operate on an entire file, it's difficult or impossible to use that utility in an otherwise asynchronous program (since it will take arbitrarily long to complete and wants to run all at once). This is probably not as serious a concern since people who aren't writing asynchronous code will almost always write things that aren't asynchronous friendly and there's nothing about this specific proposal that makes that any worse. Jean-Paul > >> And it would certainly be bad if this were the *only* interface >> Cryptography provided to certain functionality (which, fortunately, >> isn't what you're suggesting, I think). >> > > That is definitely not what I'm suggesting. Just an alternate interface for > a specific option. > > >> >> Jean-Paul >> _______________________________________________ >> Cryptography-dev mailing list >> Cryptography-dev at python.org >> https://mail.python.org/mailman/listinfo/cryptography-dev >> > > > > -- > Dr. Michael Iverson > Director of Information Technology > Hatteras Printing > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev at python.org > https://mail.python.org/mailman/listinfo/cryptography-dev From donald at stufft.io Mon Jan 19 18:10:46 2015 From: donald at stufft.io (Donald Stufft) Date: Mon, 19 Jan 2015 12:10:46 -0500 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: References: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> <9E690175-57CE-4C70-A441-7A7F287E4861@stufft.io> Message-ID: <16414730-EF6B-4FE2-A9BA-D6AE9AB02151@stufft.io> > On Jan 19, 2015, at 11:42 AM, Michael Iverson wrote: > > > > On Mon, Jan 19, 2015 at 11:27 AM, Donald Stufft > wrote: > >> I'd also be concerned about the cryptographic implications of this. I'm not sure if this is entirely correct, but it seems if you set your chunk size = AES block size, you essentially are encrypting in ECB mode. > > The actual details of what you?d need to do is more involved than just calling encrypt() with the same key on chunks. That?s just a high level ?here?s the general idea thing?. In reality you?d encrypt the stream using the streaming encryption APIs (so you?d use something like CBC or CTR) and you?d take that output and break it into chunks as well, and you?d authenticate each of those chunks. > > > > OK, I see what you're saying now. > > You would continue to operate the cipher in CBC mode, and process chunks in order. You would then HMAC each chunk, so that each could be validated prior to decryption and writing that chunk to the output. No unvalidated data would ever be passed downstream. At worst, you would have partial data. > > To handle missing or out of order chunks, you could maintain an hmac digest of all blocks processed prior to the current block, in addition to the added block. A missing block would cause the HMAC to fail. There would also need to be a method to ensure that we only a certain number of chunks. > Yea something like this. It protects as much as we can with a streaming API. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.l.kehrer at gmail.com Mon Jan 19 18:16:27 2015 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Mon, 19 Jan 2015 11:16:27 -0600 Subject: [Cryptography-dev] Create Fernet API allowing streaming encryption and decryption from file-like objects. In-Reply-To: <16414730-EF6B-4FE2-A9BA-D6AE9AB02151@stufft.io> References: <9684F8E1-F7EC-4380-ABCE-3438BFE2908F@stufft.io> <9E690175-57CE-4C70-A441-7A7F287E4861@stufft.io> <16414730-EF6B-4FE2-A9BA-D6AE9AB02151@stufft.io> Message-ID: On this subject I had some conversations with Brian Warner at Real World Cryptography about the format Tahoe-LAFS developed (merkle tree computed over segments of ciphertext). I have a vague idea that we can extract this and generalize it for a seekable/streamable encryption recipe. -------------- next part -------------- An HTML attachment was scrubbed... URL: