From awd at ddg.com  Fri Feb 16 16:16:29 2018
From: awd at ddg.com (Andrew Donoho)
Date: Fri, 16 Feb 2018 15:16:29 -0600
Subject: [Cryptography-dev] =?utf-8?q?Destroying_keys_and_secrets?=
 =?utf-8?b?4oCm?=
Message-ID: <1D23C51D-B30E-4D4D-B7C0-81490846CDB3@ddg.com>

Gentlefolk,



	Apparently, my Google-fu is weak and I come seeking advice.

	Secret management is important. In particular, I want to make sure that any secrets I decrypt are erased from memory before the storage is reclaimed by the VM. In other environments, I would just dig into each object until I get the pointer for the storage and then bang zeros, ones and randomness into the block. Then garbage collection would proceed apace. 



Here?s an example from the cryptography documentation, <https://cryptography.io/en/latest/hazmat/primitives/symmetric-encryption/>:

>>> import os
>>> from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
>>> from cryptography.hazmat.backends import default_backend
>>> backend = default_backend()
>>> key = os.urandom(32)
>>> iv = os.urandom(16)
>>> cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend)
>>> encryptor = cipher.encryptor()
>>> ct = encryptor.update(b"a secret message") + encryptor.finalize()
>>> decryptor = cipher.decryptor()
>>> decryptor.update(ct) + decryptor.finalize()
'a secret message?


The `key` above is a `bytes` object. It has storage somewhere. Even though it is a read-only Python object, I can pierce the abstraction, if I have to, with C.

My question is: has someone else already done so and published the handful of methods needed? 

If not, should this be an API added to cryptography?



Anon,
Andrew
____________________________________
Andrew W. Donoho
Donoho Design Group, L.L.C.
awd at DDG.com, +1 (512) 750-7596, twitter.com/adonoho

Doubt is not a pleasant condition, but certainty is absurd.
    ? Voltaire




From matt.s.b.42 at gmail.com  Fri Feb 16 16:54:23 2018
From: matt.s.b.42 at gmail.com (Matt Bullock)
Date: Fri, 16 Feb 2018 21:54:23 +0000
Subject: [Cryptography-dev] =?utf-8?q?Destroying_keys_and_secrets?=
	=?utf-8?b?4oCm?=
In-Reply-To: <1D23C51D-B30E-4D4D-B7C0-81490846CDB3@ddg.com>
References: <1D23C51D-B30E-4D4D-B7C0-81490846CDB3@ddg.com>
Message-ID: <CAKaOsVzB_yZPcLW7kGHvEwAkCq_UQD_0D3C+cH_jtkLBZ0aR_w@mail.gmail.com>

This is an area I've spent a fairly significant amount of time
investigating. My conclusion was "no, there is no reliable way to do this",
but I'd love to be proven wrong.

The fundamental problem is that how exactly <Python implementation> stores
variables in memory is not defined as part of the implementation
requirements, and so even if you can figure out a way to wipe a value from
memory on one implementation, there is no guarantee that it will work on
any other implementation.

With CPython, you /can/ technically use "ctypes" to read and write memory
directly, and by introspecting a similar object (say, an arbitrary string
with known value), you can reach in and overwrite data that is /probably/
the data you're trying to wipe out.

Unfortunately, this approach makes a lot of assumptions about how CPython
actually structures variables in memory: assumptions that may or may not be
valid today and may or may not be valid in the future (or the past, for
that matter). /At best/ the only thing you can reasonably say is that this
/probably/ works for the specific version of the specific implementation
you have tested it on, and that it /might/ not have adverse side effects.

I've also toyed with the idea of making a C extension analogous to the
SecretKey structure from Java, which never lets the actual key material out
of the structure, but in order to actually do anything with the key
material it still needs to surface to Python at some point, which puts us
back in the same position we started with. ..unless we were to re-implement
all of pyca/cryptography in Cython, but that's a thought for a different
time...

   --Matt

On Fri, Feb 16, 2018 at 1:25 PM Andrew Donoho <awd at ddg.com> wrote:

> Gentlefolk,
>
>
>
>         Apparently, my Google-fu is weak and I come seeking advice.
>
>         Secret management is important. In particular, I want to make sure
> that any secrets I decrypt are erased from memory before the storage is
> reclaimed by the VM. In other environments, I would just dig into each
> object until I get the pointer for the storage and then bang zeros, ones
> and randomness into the block. Then garbage collection would proceed apace.
>
>
>
> Here?s an example from the cryptography documentation, <
> https://cryptography.io/en/latest/hazmat/primitives/symmetric-encryption/
> >:
>
> >>> import os
> >>> from cryptography.hazmat.primitives.ciphers import Cipher, algorithms,
> modes
> >>> from cryptography.hazmat.backends import default_backend
> >>> backend = default_backend()
> >>> key = os.urandom(32)
> >>> iv = os.urandom(16)
> >>> cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend)
> >>> encryptor = cipher.encryptor()
> >>> ct = encryptor.update(b"a secret message") + encryptor.finalize()
> >>> decryptor = cipher.decryptor()
> >>> decryptor.update(ct) + decryptor.finalize()
> 'a secret message?
>
>
> The `key` above is a `bytes` object. It has storage somewhere. Even though
> it is a read-only Python object, I can pierce the abstraction, if I have
> to, with C.
>
> My question is: has someone else already done so and published the handful
> of methods needed?
>
> If not, should this be an API added to cryptography?
>
>
>
> Anon,
> Andrew
> ____________________________________
> Andrew W. Donoho
> Donoho Design Group, L.L.C.
> awd at DDG.com, +1 (512) 750-7596 <(512)%20750-7596>, twitter.com/adonoho
>
> Doubt is not a pleasant condition, but certainty is absurd.
>     ? Voltaire
>
>
>
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180216/0af8e3ad/attachment.html>

From me at johnpacific.com  Fri Feb 16 17:44:39 2018
From: me at johnpacific.com (John Pacific)
Date: Fri, 16 Feb 2018 15:44:39 -0700
Subject: [Cryptography-dev] =?utf-8?q?Destroying_keys_and_secrets?=
	=?utf-8?b?4oCm?=
In-Reply-To: <CAKaOsVzB_yZPcLW7kGHvEwAkCq_UQD_0D3C+cH_jtkLBZ0aR_w@mail.gmail.com>
References: <1D23C51D-B30E-4D4D-B7C0-81490846CDB3@ddg.com>
 <CAKaOsVzB_yZPcLW7kGHvEwAkCq_UQD_0D3C+cH_jtkLBZ0aR_w@mail.gmail.com>
Message-ID: <CAHr1PUP6Wvzdho02YfoZg4mWttdz3CXRkPjssBUJ1wtLuPUBYQ@mail.gmail.com>

Afaik, there is no reliable way to do this in Python.

I have taken some time to implement some OpenSSL methods that at least
clear the memory on private numbers before freeing it in BIGNUMBER
operations.

See the PR here: https://github.com/pyca/cryptography/pull/4072

However, as it comes for the Elliptic curve

On Feb 16, 2018 14:54, "Matt Bullock" <matt.s.b.42 at gmail.com> wrote:

> This is an area I've spent a fairly significant amount of time
> investigating. My conclusion was "no, there is no reliable way to do this",
> but I'd love to be proven wrong.
>
> The fundamental problem is that how exactly <Python implementation> stores
> variables in memory is not defined as part of the implementation
> requirements, and so even if you can figure out a way to wipe a value from
> memory on one implementation, there is no guarantee that it will work on
> any other implementation.
>
> With CPython, you /can/ technically use "ctypes" to read and write memory
> directly, and by introspecting a similar object (say, an arbitrary string
> with known value), you can reach in and overwrite data that is /probably/
> the data you're trying to wipe out.
>
> Unfortunately, this approach makes a lot of assumptions about how CPython
> actually structures variables in memory: assumptions that may or may not be
> valid today and may or may not be valid in the future (or the past, for
> that matter). /At best/ the only thing you can reasonably say is that this
> /probably/ works for the specific version of the specific implementation
> you have tested it on, and that it /might/ not have adverse side effects.
>
> I've also toyed with the idea of making a C extension analogous to the
> SecretKey structure from Java, which never lets the actual key material out
> of the structure, but in order to actually do anything with the key
> material it still needs to surface to Python at some point, which puts us
> back in the same position we started with. ..unless we were to re-implement
> all of pyca/cryptography in Cython, but that's a thought for a different
> time...
>
>    --Matt
>
> On Fri, Feb 16, 2018 at 1:25 PM Andrew Donoho <awd at ddg.com> wrote:
>
>> Gentlefolk,
>>
>>
>>
>>         Apparently, my Google-fu is weak and I come seeking advice.
>>
>>         Secret management is important. In particular, I want to make
>> sure that any secrets I decrypt are erased from memory before the storage
>> is reclaimed by the VM. In other environments, I would just dig into each
>> object until I get the pointer for the storage and then bang zeros, ones
>> and randomness into the block. Then garbage collection would proceed apace.
>>
>>
>>
>> Here?s an example from the cryptography documentation, <
>> https://cryptography.io/en/latest/hazmat/primitives/symmetric-encryption/
>> >:
>>
>> >>> import os
>> >>> from cryptography.hazmat.primitives.ciphers import Cipher,
>> algorithms, modes
>> >>> from cryptography.hazmat.backends import default_backend
>> >>> backend = default_backend()
>> >>> key = os.urandom(32)
>> >>> iv = os.urandom(16)
>> >>> cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend)
>> >>> encryptor = cipher.encryptor()
>> >>> ct = encryptor.update(b"a secret message") + encryptor.finalize()
>> >>> decryptor = cipher.decryptor()
>> >>> decryptor.update(ct) + decryptor.finalize()
>> 'a secret message?
>>
>>
>> The `key` above is a `bytes` object. It has storage somewhere. Even
>> though it is a read-only Python object, I can pierce the abstraction, if I
>> have to, with C.
>>
>> My question is: has someone else already done so and published the
>> handful of methods needed?
>>
>> If not, should this be an API added to cryptography?
>>
>>
>>
>> Anon,
>> Andrew
>> ____________________________________
>> Andrew W. Donoho
>> Donoho Design Group, L.L.C.
>> awd at DDG.com, +1 (512) 750-7596 <(512)%20750-7596>, twitter.com/adonoho
>>
>> Doubt is not a pleasant condition, but certainty is absurd.
>>     ? Voltaire
>>
>>
>>
>> _______________________________________________
>> Cryptography-dev mailing list
>> Cryptography-dev at python.org
>> https://mail.python.org/mailman/listinfo/cryptography-dev
>>
>
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180216/103660bc/attachment-0001.html>

From python-cryptography at qu3.org  Fri Feb 16 22:04:00 2018
From: python-cryptography at qu3.org (Alex)
Date: Sat, 17 Feb 2018 04:04:00 +0100
Subject: [Cryptography-dev] =?utf-8?q?Destroying_keys_and_secrets?=
 =?utf-8?b?4oCm?=
In-Reply-To: <1D23C51D-B30E-4D4D-B7C0-81490846CDB3@ddg.com>
References: <1D23C51D-B30E-4D4D-B7C0-81490846CDB3@ddg.com>
Message-ID: <d057417f-ce19-1a60-4d18-18680c56279a@nigl.biz>

You might have a look at:
https://github.com/dnet/pysecstr

Here is a longer description about the used method:
https://www.sjoerdlangkemper.nl/2016/06/09/clearing-memory-in-python/

Disclaimer: This is not a recommendation just a hint for further
exploration. Be also aware that you can't control if and when Python
swaps memory to disk.

Alex

On 16.02.2018 22:16, Andrew Donoho wrote:
> Gentlefolk,
> 
> 
> 
> 	Apparently, my Google-fu is weak and I come seeking advice.
> 
> 	Secret management is important. In particular, I want to make sure that any secrets I decrypt are erased from memory before the storage is reclaimed by the VM. In other environments, I would just dig into each object until I get the pointer for the storage and then bang zeros, ones and randomness into the block. Then garbage collection would proceed apace. 
> 
> 
> 
> Here?s an example from the cryptography documentation, <https://cryptography.io/en/latest/hazmat/primitives/symmetric-encryption/>:
> 
>>>> import os
>>>> from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
>>>> from cryptography.hazmat.backends import default_backend
>>>> backend = default_backend()
>>>> key = os.urandom(32)
>>>> iv = os.urandom(16)
>>>> cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend)
>>>> encryptor = cipher.encryptor()
>>>> ct = encryptor.update(b"a secret message") + encryptor.finalize()
>>>> decryptor = cipher.decryptor()
>>>> decryptor.update(ct) + decryptor.finalize()
> 'a secret message?
> 
> 
> The `key` above is a `bytes` object. It has storage somewhere. Even though it is a read-only Python object, I can pierce the abstraction, if I have to, with C.
> 
> My question is: has someone else already done so and published the handful of methods needed? 
> 
> If not, should this be an API added to cryptography?
> 
> 
> 
> Anon,
> Andrew
> ____________________________________
> Andrew W. Donoho
> Donoho Design Group, L.L.C.
> awd at DDG.com, +1 (512) 750-7596, twitter.com/adonoho
> 
> Doubt is not a pleasant condition, but certainty is absurd.
>     ? Voltaire
> 
> 
> 
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3992 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180217/a7d5d56d/attachment.bin>

From andrew.donoho at gmail.com  Mon Feb 19 15:21:08 2018
From: andrew.donoho at gmail.com (Andrew W. Donoho)
Date: Mon, 19 Feb 2018 14:21:08 -0600
Subject: [Cryptography-dev] Cryptography-dev Digest, Vol 54, Issue 2
In-Reply-To: <mailman.50.1518886803.7409.cryptography-dev@python.org>
References: <mailman.50.1518886803.7409.cryptography-dev@python.org>
Message-ID: <2DA8FA6A-2E7D-48F5-BC99-F3DD99DF1C13@gmail.com>



> On Feb 17, 2018, at 11:00 , cryptography-dev-request at python.org wrote:
> 
> Date: Sat, 17 Feb 2018 04:04:00 +0100
> From: Alex <python-cryptography at qu3.org>
> To: cryptography-dev at python.org
> Subject: Re: [Cryptography-dev] Destroying keys and secrets?
> Message-ID: <d057417f-ce19-1a60-4d18-18680c56279a at nigl.biz>
> Content-Type: text/plain; charset="utf-8"
> 
> You might have a look at:
> https://github.com/dnet/pysecstr
> 
> Here is a longer description about the used method:
> https://www.sjoerdlangkemper.nl/2016/06/09/clearing-memory-in-python/
> 
> Disclaimer: This is not a recommendation just a hint for further
> exploration. Be also aware that you can't control if and when Python
> swaps memory to disk.



> On Feb 16, 2018, at 16:44 , cryptography-dev-request at python.org wrote:
> Date: Fri, 16 Feb 2018 21:54:23 +0000
> From: Matt Bullock <matt.s.b.42 at gmail.com>
> To: cryptography-dev at python.org
> Subject: Re: [Cryptography-dev] Destroying keys and secrets?
> 
> I've also toyed with the idea of making a C extension analogous to the
> SecretKey structure from Java, which never lets the actual key material out
> of the structure, but in order to actually do anything with the key
> material it still needs to surface to Python at some point, which puts us
> back in the same position we started with. ..unless we were to re-implement
> all of pyca/cryptography in Cython, but that's a thought for a different
> time?



> On Feb 16, 2018, at 16:44 , cryptography-dev-request at python.org wrote:
> 
> Date: Fri, 16 Feb 2018 15:44:39 -0700
> From: John Pacific <me at johnpacific.com>
> To: cryptography-dev at python.org
> Subject: Re: [Cryptography-dev] Destroying keys and secrets?
> Message-ID:
> 	<CAHr1PUP6Wvzdho02YfoZg4mWttdz3CXRkPjssBUJ1wtLuPUBYQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Afaik, there is no reliable way to do this in Python.
> 
> I have taken some time to implement some OpenSSL methods that at least
> clear the memory on private numbers before freeing it in BIGNUMBER
> operations.
> 
> See the PR here: https://github.com/pyca/cryptography/pull/4072
> 
> However, as it comes for the Elliptic curve





Gentlemen,



	Thank you for your fine answers. They comport with my expectations.

	Without boring you with my details, I am doing something similar to an ephemeral ECDH exchange. The secrets have a very short lifetime. Hence, I?m not worried about swapping to disk. (Though most of the cloud vendors claim the swap vm is encrypted.)

For example:

secret_bytes = os.urandom(384//8)
secret_int = int.from_bytes(secret_bytes, byteorder='big')
private_key = ec.derive_private_key(secret_int, ec.SECP384R1(), backend)



	I would like secret_bytes, secret_int, and private_key to be zeroed-out before I exit the function. I want as precise an object lifetime as possible. While I fully understand the values of a library maintainer wishing to have a robust solution for all supported versions and runtimes, I have explicit control of this function and its execution environment. I?m more concerned about revealing the secret in a hostile cloud. In particular, I don?t have the string aliasing problem. Yes, this is a fragile and extremely hazardous solution. OTOH, the industry just freaked out about Meltdown. It behooves us all to manage key lifetimes carefully. 



	As I dig into examining this problem further, I assume the ec.derive_private_key() function takes a conservative approach and doesn?t ever copy the secret_int? If I?m living well, banging the bytes in secret_bytes might hit the only copy of the secret. Depends upon whether int.from_bytes() copies the storage of an immutable source.




	Again, thank you for your answers. I?ll keep you posted with my progress.



Anon,
Andrew
____________________________________
Andrew W. Donoho
Donoho Design Group, L.L.C.
andrew.donoho at gmail.com, +1 (512) 666-7596, twitter.com/adonoho

No risk, no art.
	No art, no reward.
		-- Seth Godin



From paul.l.kehrer at gmail.com  Mon Feb 19 20:14:25 2018
From: paul.l.kehrer at gmail.com (Paul Kehrer)
Date: Mon, 19 Feb 2018 17:14:25 -0800
Subject: [Cryptography-dev] Cryptography-dev Digest, Vol 54, Issue 2
In-Reply-To: <2DA8FA6A-2E7D-48F5-BC99-F3DD99DF1C13@gmail.com>
References: <mailman.50.1518886803.7409.cryptography-dev@python.org>
 <2DA8FA6A-2E7D-48F5-BC99-F3DD99DF1C13@gmail.com>
Message-ID: <CABj5TKTv8cP6XGdCrABxF6qXJNZoipufe6_xy2y_N+R95KSuTA@mail.gmail.com>

On February 20, 2018 at 4:21:32 AM, Andrew W. Donoho (
andrew.donoho at gmail.com) wrote:


Gentlemen,



Thank you for your fine answers. They comport with my expectations.

Without boring you with my details, I am doing something similar to an
ephemeral ECDH exchange. The secrets have a very short lifetime. Hence, I?m
not worried about swapping to disk. (Though most of the cloud vendors claim
the swap vm is encrypted.)

For example:

secret_bytes = os.urandom(384//8)
secret_int = int.from_bytes(secret_bytes, byteorder='big')
private_key = ec.derive_private_key(secret_int, ec.SECP384R1(), backend)



I would like secret_bytes, secret_int, and private_key to be zeroed-out
before I exit the function. I want as precise an object lifetime as
possible. While I fully understand the values of a library maintainer
wishing to have a robust solution for all supported versions and runtimes,
I have explicit control of this function and its execution environment. I?m
more concerned about revealing the secret in a hostile cloud. In
particular, I don?t have the string aliasing problem. Yes, this is a
fragile and extremely hazardous solution. OTOH, the industry just freaked
out about Meltdown. It behooves us all to manage key lifetimes carefully.



As I dig into examining this problem further, I assume the
ec.derive_private_key() function takes a conservative approach and doesn?t
ever copy the secret_int? If I?m living well, banging the bytes in
secret_bytes might hit the only copy of the secret. Depends upon whether
int.from_bytes() copies the storage of an immutable source.



Access to Python's memory (via side channel or dumping as root) is not part
of pyca/cryptography's threat model at this time so we don't attempt to
protect against it. Making it part of our threat model would be difficult
due in part to the reasons you stated above as well as the difficulty in
writing tests to prevent regression, but let's talk about what CPython does
in this case.

int.from_bytes will unfortunately make a copy (to a Python integer). That
int will then be copied into a BN via _int_to_bn (
https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/backend.py#L317-L346)
when you call derive_private_key. It will actually be converted twice (a
thing we should fix) (
https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/backend.py#L1383-L1419).
Although the resulting BNs will themselves be zeroed as freed, this means a
secret scalar bytestring created in Python will be resident in memory no
less than 5 times (3 byte strings, 2 numbers).

Obviously the next logical question is why you'd provide a Python integer
when we're just going to convert it back to big endian bytes anyway.
Disregarding the memory clearing issue it's also inefficient. When
originally designing some of the APIs we made a mistake and chose integers
instead of big endian bytes (see: numbers classes). We have not yet added
alternate APIs to potentially enable us to deprecate numbers because the
improvement in efficiency probably isn't worth the pain of trying to
convert the huge number of users of those classes.

ec.derive_private_key_from_bytes(secret_bytes, ec.SECP384R1(), backend)
could potentially be a way to do this specific operation while reducing the
number of copies (to zero in Python and 2-3 in OpenSSL, although the latter
are zeroed), but without tests that can detect non-required copies of
secret material it would be extremely hard to prevent regression in the
long term as the code is updated.

Given your chosen constraints have you considered deriving a key in
subprocess, serializing it, and reading it from stdout in the parent
process? By doing this you'd have a precisely defined intermediate object
lifetime and the only secret in the parent process's memory would be a
single DER or PEM bytestring containing the EC key.

-Paul Kehrer (reaperhulk)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180219/d9fb3699/attachment.html>

From andrew.donoho at gmail.com  Wed Feb 21 10:51:19 2018
From: andrew.donoho at gmail.com (Andrew W. Donoho)
Date: Wed, 21 Feb 2018 09:51:19 -0600
Subject: [Cryptography-dev] Destroying keys and secrets?
In-Reply-To: <mailman.67.1519146004.25271.cryptography-dev@python.org>
References: <mailman.67.1519146004.25271.cryptography-dev@python.org>
Message-ID: <BD30276E-12DA-44D9-A977-9B17DBAA8EAB@gmail.com>



> On Feb 20, 2018, at 11:00 , cryptography-dev-request at python.org wrote:
> Date: Mon, 19 Feb 2018 17:14:25 -0800
> From: Paul Kehrer <paul.l.kehrer at gmail.com>
> To: cryptography-dev at python.org
> Subject: Re: [Cryptography-dev] Cryptography-dev Digest, Vol 54, Issue
> 	2
> Message-ID:
> 	<CABj5TKTv8cP6XGdCrABxF6qXJNZoipufe6_xy2y_N+R95KSuTA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8?
> 
> Access to Python's memory (via side channel or dumping as root) is not part
> of pyca/cryptography's threat model at this time so we don't attempt to
> protect against it. Making it part of our threat model would be difficult
> due in part to the reasons you stated above as well as the difficulty in
> writing tests to prevent regression, but let's talk about what CPython does
> in this case.



Paul,



	Thank you for your enlightening comments above and below.



What Am I Really Trying To Do?:

This function is initially an AWS lambda function that I also intend to support on Google Cloud and MS Azure. This august group of engineers obviously knows of my concern but I nonetheless wish to emphasize the problem. Perhaps I can also contribute to its solution? While I have control of the function?s environment, I don?t have total control. For example, while I can hope the host OS has enough entropy in its pool to support os.urandom(), the AWS recommendation, with which I concur, is that I get my random_bytes from the AWS KMS service and stir it up with an HKDF. As in:



random_bytes = os.urandom(512//8)  # <= Should come from AWS KMS random bytes service.

# Derive a SECP384R1 private key using SHA-512, _salt512, and guid.
hkdf = HKDF(algorithm=sha512, length=384//8, salt=_salt512, info=guid, backend=backend)
secret_bytes = hkdf.derive(random_bytes)
secret_int = int.from_bytes(secret_bytes, byteorder='big')
private_key = ec.derive_private_key(secret_int, ec.SECP384R1(), backend)



As I understand it, each of the cloud vendors offers a high entropy source of random bytes. Upon completion of this function, I want to scrub RAM of key material. I will never get another chance to address the existence of this key material. There is no goodbye kiss from the cloud function execution environments. These are one-shot function calls. If I am to ensure that this crypto toxic waste doesn?t come back to bite my service, then I must dispose of it in the function context that created it. 



In other words, pyca/cryptography is likely to be used in a much more dynamic environment than heretofore. Considering that pyca/cryptography has largely succeeded in building a civilized interface to crypto routines, helping folks implement good secret/key hygiene seems in scope for the project?s goals. Considering Amazon?s embrace of pyca/cryptography for their Python lambda functions and AWS Encryption SDK, I am unlikely to be the sole user.



> int.from_bytes will unfortunately make a copy (to a Python integer). That
> int will then be copied into a BN via _int_to_bn (
> https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/backend.py#L317-L346)
> when you call derive_private_key. It will actually be converted twice (a
> thing we should fix) (
> https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/backend.py#L1383-L1419).
> Although the resulting BNs will themselves be zeroed as freed, this means a
> secret scalar bytestring created in Python will be resident in memory no
> less than 5 times (3 byte strings, 2 numbers).
> 
> Obviously the next logical question is why you'd provide a Python integer
> when we're just going to convert it back to big endian bytes anyway.
> Disregarding the memory clearing issue it's also inefficient. When
> originally designing some of the APIs we made a mistake and chose integers
> instead of big endian bytes (see: numbers classes). We have not yet added
> alternate APIs to potentially enable us to deprecate numbers because the
> improvement in efficiency probably isn't worth the pain of trying to
> convert the huge number of users of those classes.
> 
> ec.derive_private_key_from_bytes(secret_bytes, ec.SECP384R1(), backend)
> could potentially be a way to do this specific operation while reducing the
> number of copies (to zero in Python and 2-3 in OpenSSL, although the latter
> are zeroed), but without tests that can detect non-required copies of
> secret material it would be extremely hard to prevent regression in the
> long term as the code is updated.



	Thank you for the above excellent exposition of the state of data copying in pyca/cryptography. Considering the appropriate "st?rm und drang" that accompanied Meltdown, Specter, and other exploits around key material, I think many developers have an interest in making the changes to their code that allows them to ensure precise key material lifetimes. Of course, with modern internet survey tools, the pyca/cryptography team can easily ask their users if they would make these changes.

	To your point about regression testing the copying of key material: I place this in the category of expertise of the library developer. There are plenty of similar cryptographic issues that the pyca/cryptography team has had to tackle through discipline and intra-project communication. Obviously, because your team already clears and then frees items passed to and from OpenSSL, you know how to do this. How can I help?




	I also note that ec.derive_private_key() has a paired routine where pyca/cryptography takes responsibility for the secret_bytes: ec.generate_private_key(). If it manages its secret copies carefully, I might consider using it instead of providing my own secret_bytes. It would, of course, depend upon pyca/cryptography?s high entropy data source (os.urandom()?). As this is in the hazmat section of the library, perhaps a discussion in the documentation about digital toxic/hazardous waste is appropriate? How can I help?



> Given your chosen constraints have you considered deriving a key in
> subprocess, serializing it, and reading it from stdout in the parent
> process? By doing this you'd have a precisely defined intermediate object
> lifetime and the only secret in the parent process's memory would be a
> single DER or PEM bytestring containing the EC key.



	That is one excellent way to achieve my goal in a normal execution environment. All of the industry trends lead me to believe that AWS lambda function-like environments will become a high percentage of where crypto operations are performed. Due to their small scope and statelessness, they lend themselves to focussed and highly tuned solutions. They are a natural place for the cryptographic core functions of many systems to reside. Put all of your keys in one basket and then watch that basket very carefully.



	Thank you for taking the time to examine my concerns. 



Anon,
Andrew
____________________________________
Andrew W. Donoho
Donoho Design Group, L.L.C.
andrew.donoho at gmail.com, +1 (512) 666-7596, twitter.com/adonoho

No risk, no art.
	No art, no reward.
		-- Seth Godin



From andrew.donoho at gmail.com  Thu Feb 22 16:41:02 2018
From: andrew.donoho at gmail.com (Andrew W. Donoho)
Date: Thu, 22 Feb 2018 15:41:02 -0600
Subject: [Cryptography-dev] Destroying keys and secrets?
In-Reply-To: <mailman.67.1519146004.25271.cryptography-dev@python.org>
References: <mailman.67.1519146004.25271.cryptography-dev@python.org>
Message-ID: <66904CA1-015C-4181-B805-0D811AF250AE@gmail.com>



> On Feb 20, 2018, at 11:00 , cryptography-dev-request at python.org wrote:
> 
> ec.derive_private_key_from_bytes(secret_bytes, ec.SECP384R1(), backend)
> could potentially be a way to do this specific operation while reducing the
> number of copies (to zero in Python and 2-3 in OpenSSL, although the latter
> are zeroed), but without tests that can detect non-required copies of
> secret material it would be extremely hard to prevent regression in the
> long term as the code is updated.




Paul,



	Based upon your hint above I just went in to the code and looked around. It looks like a straightforward extension. Note: I am not a library developer and, hence, have not developed all of the various skills to properly build pyca/cryptography. Nor am I particularly knowledgable about Python v2.7 & v3.6 interoperation issues, much less CPython, Cython and PyPy interoperation. The below code is a gedanken exploration to see how hard or involved it might actually be to extend pyca/cryptography. 

	TL;DR: It isn?t hard. I C&P three functions and then modded them. Gratz to the team on an excellent design.

	It is clear that I could deploy my own copy of cryptography and call it a day. But I believe that key hygiene is an important social good. I am happy to help look at/write each routine that imports key material and propose versions that allow good key hygiene. As you note above, there is no solution for your regression testing issue except to discuss it in code commentary. Now to find a routine that can bang the contents of a bytes array to 0, 1 and then random.

	Should I proceed to engage with your team or just continue on my merry way? I want to help but if it isn?t something the team prioritizes highly right now, I fully understand.



Anon,
Andrew
____________________________________
Andrew W. Donoho
Donoho Design Group, L.L.C.
andrew.donoho at gmail.com, +1 (512) 666-7596, twitter.com/adonoho

No risk, no art.
	No art, no reward.
		-- Seth Godin




===== backend.py =====

def _bytes_to_bn(self, num, bn=None):
    # Was: def _int_to_bn(self, num, bn=None):
    """
    Converts a python bytes array to a BIGNUM. The returned BIGNUM will not
    be garbage collected (to support adding them to structs that take
    ownership of the object). Be sure to register it for GC if it will
    be discarded after use.
    """
    assert bn is None or bn != self._ffi.NULL

    if bn is None:
        bn = self._ffi.NULL

    bn_ptr = self._lib.BN_bin2bn(num, len(num), bn)
    self.openssl_assert(bn_ptr != self._ffi.NULL)
    return bn_ptr


def derive_elliptic_curve_private_key_bytes(self, private_bytes, curve):
    # Was: def derive_elliptic_curve_private_key(self, private_value, curve):
    curve_nid = self._elliptic_curve_to_nid(curve)

    ec_cdata = self._lib.EC_KEY_new_by_curve_name(curve_nid)
    self.openssl_assert(ec_cdata != self._ffi.NULL)
    ec_cdata = self._ffi.gc(ec_cdata, self._lib.EC_KEY_free)

    get_func, group = self._ec_key_determine_group_get_func(ec_cdata)

    point = self._lib.EC_POINT_new(group)
    self.openssl_assert(point != self._ffi.NULL)
    point = self._ffi.gc(point, self._lib.EC_POINT_free)

    value = self._bytes_to_bn(private_bytes)
    value = self._ffi.gc(value, self._lib.BN_clear_free)

    with self._tmp_bn_ctx() as bn_ctx:
        res = self._lib.EC_POINT_mul(group, point, value, self._ffi.NULL,
                                     self._ffi.NULL, bn_ctx)
        self.openssl_assert(res == 1)

        bn_x = self._lib.BN_CTX_get(bn_ctx)
        bn_y = self._lib.BN_CTX_get(bn_ctx)

        res = get_func(group, point, bn_x, bn_y, bn_ctx)
        self.openssl_assert(res == 1)

    res = self._lib.EC_KEY_set_public_key(ec_cdata, point)
    self.openssl_assert(res == 1)
    private = self._bytes_to_bn(private_bytes)
    private = self._ffi.gc(private, self._lib.BN_clear_free)
    res = self._lib.EC_KEY_set_private_key(ec_cdata, private)
    self.openssl_assert(res == 1)

    evp_pkey = self._ec_cdata_to_evp_pkey(ec_cdata)

    return _EllipticCurvePrivateKey(self, ec_cdata, evp_pkey)



===== ec.py =====

def derive_private_key_bytes(private_value, curve, backend):
    # Was: def derive_private_key(private_value, curve, backend):
   if not isinstance(curve, EllipticCurve):
        raise TypeError("curve must provide the EllipticCurve interface.")

    return backend.derive_elliptic_curve_private_key_bytes(private_value, curve)





From me at johnpacific.com  Thu Feb 22 16:54:07 2018
From: me at johnpacific.com (John Pacific)
Date: Thu, 22 Feb 2018 14:54:07 -0700
Subject: [Cryptography-dev] Destroying keys and secrets?
In-Reply-To: <66904CA1-015C-4181-B805-0D811AF250AE@gmail.com>
References: <mailman.67.1519146004.25271.cryptography-dev@python.org>
 <66904CA1-015C-4181-B805-0D811AF250AE@gmail.com>
Message-ID: <CAHr1PUO9gLy_9v13zdJEZ15R+o7WfEaJo2Qw9tqUP7xQTFV99Q@mail.gmail.com>

Andrew,

If you notice the call to `BN_clear_free`, it zeros the memory of the key
once it gets garbage collected.

However, as several others have explained, this does not prevent the memory
getting stored on the disk in some manner.

-tux

On Feb 22, 2018 14:41, "Andrew W. Donoho" <andrew.donoho at gmail.com> wrote:

>
>
> > On Feb 20, 2018, at 11:00 , cryptography-dev-request at python.org wrote:
> >
> > ec.derive_private_key_from_bytes(secret_bytes, ec.SECP384R1(), backend)
> > could potentially be a way to do this specific operation while reducing
> the
> > number of copies (to zero in Python and 2-3 in OpenSSL, although the
> latter
> > are zeroed), but without tests that can detect non-required copies of
> > secret material it would be extremely hard to prevent regression in the
> > long term as the code is updated.
>
>
>
>
> Paul,
>
>
>
>         Based upon your hint above I just went in to the code and looked
> around. It looks like a straightforward extension. Note: I am not a library
> developer and, hence, have not developed all of the various skills to
> properly build pyca/cryptography. Nor am I particularly knowledgable about
> Python v2.7 & v3.6 interoperation issues, much less CPython, Cython and
> PyPy interoperation. The below code is a gedanken exploration to see how
> hard or involved it might actually be to extend pyca/cryptography.
>
>         TL;DR: It isn?t hard. I C&P three functions and then modded them.
> Gratz to the team on an excellent design.
>
>         It is clear that I could deploy my own copy of cryptography and
> call it a day. But I believe that key hygiene is an important social good.
> I am happy to help look at/write each routine that imports key material and
> propose versions that allow good key hygiene. As you note above, there is
> no solution for your regression testing issue except to discuss it in code
> commentary. Now to find a routine that can bang the contents of a bytes
> array to 0, 1 and then random.
>
>         Should I proceed to engage with your team or just continue on my
> merry way? I want to help but if it isn?t something the team prioritizes
> highly right now, I fully understand.
>
>
>
> Anon,
> Andrew
> ____________________________________
> Andrew W. Donoho
> Donoho Design Group, L.L.C.
> andrew.donoho at gmail.com, +1 (512) 666-7596, twitter.com/adonoho
>
> No risk, no art.
>         No art, no reward.
>                 -- Seth Godin
>
>
>
>
> ===== backend.py =====
>
> def _bytes_to_bn(self, num, bn=None):
>     # Was: def _int_to_bn(self, num, bn=None):
>     """
>     Converts a python bytes array to a BIGNUM. The returned BIGNUM will not
>     be garbage collected (to support adding them to structs that take
>     ownership of the object). Be sure to register it for GC if it will
>     be discarded after use.
>     """
>     assert bn is None or bn != self._ffi.NULL
>
>     if bn is None:
>         bn = self._ffi.NULL
>
>     bn_ptr = self._lib.BN_bin2bn(num, len(num), bn)
>     self.openssl_assert(bn_ptr != self._ffi.NULL)
>     return bn_ptr
>
>
> def derive_elliptic_curve_private_key_bytes(self, private_bytes, curve):
>     # Was: def derive_elliptic_curve_private_key(self, private_value,
> curve):
>     curve_nid = self._elliptic_curve_to_nid(curve)
>
>     ec_cdata = self._lib.EC_KEY_new_by_curve_name(curve_nid)
>     self.openssl_assert(ec_cdata != self._ffi.NULL)
>     ec_cdata = self._ffi.gc(ec_cdata, self._lib.EC_KEY_free)
>
>     get_func, group = self._ec_key_determine_group_get_func(ec_cdata)
>
>     point = self._lib.EC_POINT_new(group)
>     self.openssl_assert(point != self._ffi.NULL)
>     point = self._ffi.gc(point, self._lib.EC_POINT_free)
>
>     value = self._bytes_to_bn(private_bytes)
>     value = self._ffi.gc(value, self._lib.BN_clear_free)
>
>     with self._tmp_bn_ctx() as bn_ctx:
>         res = self._lib.EC_POINT_mul(group, point, value, self._ffi.NULL,
>                                      self._ffi.NULL, bn_ctx)
>         self.openssl_assert(res == 1)
>
>         bn_x = self._lib.BN_CTX_get(bn_ctx)
>         bn_y = self._lib.BN_CTX_get(bn_ctx)
>
>         res = get_func(group, point, bn_x, bn_y, bn_ctx)
>         self.openssl_assert(res == 1)
>
>     res = self._lib.EC_KEY_set_public_key(ec_cdata, point)
>     self.openssl_assert(res == 1)
>     private = self._bytes_to_bn(private_bytes)
>     private = self._ffi.gc(private, self._lib.BN_clear_free)
>     res = self._lib.EC_KEY_set_private_key(ec_cdata, private)
>     self.openssl_assert(res == 1)
>
>     evp_pkey = self._ec_cdata_to_evp_pkey(ec_cdata)
>
>     return _EllipticCurvePrivateKey(self, ec_cdata, evp_pkey)
>
>
>
> ===== ec.py =====
>
> def derive_private_key_bytes(private_value, curve, backend):
>     # Was: def derive_private_key(private_value, curve, backend):
>    if not isinstance(curve, EllipticCurve):
>         raise TypeError("curve must provide the EllipticCurve interface.")
>
>     return backend.derive_elliptic_curve_private_key_bytes(private_value,
> curve)
>
>
>
>
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180222/b7e87baf/attachment.html>

From andrew.donoho at gmail.com  Fri Feb 23 13:29:05 2018
From: andrew.donoho at gmail.com (Andrew W. Donoho)
Date: Fri, 23 Feb 2018 12:29:05 -0600
Subject: [Cryptography-dev] Cryptography-dev Digest, Vol 54, Issue 5
In-Reply-To: <mailman.62.1519405204.13088.cryptography-dev@python.org>
References: <mailman.62.1519405204.13088.cryptography-dev@python.org>
Message-ID: <BCF2C440-06C6-479B-89C7-C5549513FEE0@gmail.com>



> On Feb 23, 2018, at 11:00 , cryptography-dev-request at python.org wrote:
> 
> However, as several others have explained, this does not prevent the memory
> getting stored on the disk in some manner.



tux,



	You are absolutely correct. Prudence is required. Your advice to skeptically believe that I have fully protected myself from key snooping is noted and embraced. Note that I am attempting to make the key's exposure last for as short a time as possible. Certainly you support this goal?

	I ask each cloud vendor what their VM swap encryption policy is. Some claim that just their disks are encrypted. That encrypted swap is not necessary. These are also the same folks who believe secrets are safe as environment variables. (*cough* Docker *cough* Heroku *cough*) At least AWS is now offering explicitly secure secret parameters. Yes, we should know exactly what are the swap and encryption policies of each of the function execution environment vendors. When pressed at conferences like OSCON, these vendors fall back upon the ?Top Men? argument. They employ ?Top Men? who never make mistakes. (Nor will the org stand behind them not making mistakes.) 

	BTW, this is the same ?Top Men? defense offered by AWS when I ask to see the results of any third party security audit of Boto3, the AWS access library. I hope their ?Top Men? are truly ?Top Men.? More transparency would be welcome. In particular, I?ve inquired of the AWS Encryption SDK team how they control the lifetime of their cached, unencrypted credentials. They did not have a good solution nor did they explicitly address exposed key lifetimes. I believe a member of that team has already participated in this thread. For which I thank him. Here?s a critical patch notice: Amazon Linux AMI Security Advisory: ALAS-2018-939, <https://alas.aws.amazon.com/ALAS-2018-939.html <https://alas.aws.amazon.com/ALAS-2018-939.html>>. Clearly, this threat of cross process snooping is taken seriously by AWS and, I assume, will soon be addressed in their other services. One assumes this would be done relatively quickly for the AWS Encryption SDK. I assume each of the cloud vendors views Meltdown with similar concern. I, as a user, have a responsibility too. I should not leave unencrypted key material lying around my address space. That is why I use specialized services that exploit hardware security modules. 

	Hence, I?ve come here to address the exposed key problem myself instead of depending upon the AWS Encryption SDK. This open and engaged community has rewarded my effort. Thank you.

	More folks should be asking for public disclosure of these important security policies.

	In my case, I am attempting to precisely control the lifetime of these secrets. I am attempting to minimize the time I am exposed. With the advice and guidance of this team, I feel I can now track the explicit lifetime and disposition of my EC and AES secrets. I?ll leave password strings to another time. BTW, gc.collect(generation=0) has surprisingly little impact on my test routine?s execution speeds. It appears to properly cause the collection of my ec.privateKey. In other words, once again, the pyca/cryptography team has implemented their system well. Bravo!



Anon,
Andrew
____________________________________
Andrew W. Donoho
Donoho Design Group, L.L.C.
andrew.donoho at gmail.com, +1 (512) 666-7596, twitter.com/adonoho

No risk, no art.
	No art, no reward.
		-- Seth Godin


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180223/7753ba4e/attachment.html>

From readthedocs at readthedocs.org  Fri Feb 23 15:58:19 2018
From: readthedocs at readthedocs.org (Read the Docs)
Date: Fri, 23 Feb 2018 20:58:19 -0000
Subject: [Cryptography-dev] Failed: Cryptography (404496ae)
Message-ID: <20180223205819.32538.3883@web01.servers.readthedocs.org>


Build Failed for Cryptography (latest)



You can find out more about this failure here:
https://readthedocs.org/projects/cryptography/builds/6795481/

If you have questions, a good place to start is the FAQ:
https://docs.readthedocs.org/en/latest/faq.html



Keep documenting,
Read the Docs
--
http://readthedocs.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180223/e5cb552c/attachment.html>

From readthedocs at readthedocs.org  Sat Feb 24 22:23:00 2018
From: readthedocs at readthedocs.org (Read the Docs)
Date: Sun, 25 Feb 2018 03:23:00 -0000
Subject: [Cryptography-dev] Failed: Cryptography (8374206d)
Message-ID: <20180225032300.32538.63923@web01.servers.readthedocs.org>


Build Failed for Cryptography (latest)



You can find out more about this failure here:
https://readthedocs.org/projects/cryptography/builds/6799808/

If you have questions, a good place to start is the FAQ:
https://docs.readthedocs.org/en/latest/faq.html



Keep documenting,
Read the Docs
--
http://readthedocs.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180225/2e8871f5/attachment.html>

From readthedocs at readthedocs.org  Sun Feb 25 10:18:36 2018
From: readthedocs at readthedocs.org (Read the Docs)
Date: Sun, 25 Feb 2018 15:18:36 -0000
Subject: [Cryptography-dev] Failed: Cryptography (96787f95)
Message-ID: <20180225151836.7609.89802@web05.servers.readthedocs.org>


Build Failed for Cryptography (latest)



You can find out more about this failure here:
https://readthedocs.org/projects/cryptography/builds/6801082/

If you have questions, a good place to start is the FAQ:
https://docs.readthedocs.org/en/latest/faq.html



Keep documenting,
Read the Docs
--
http://readthedocs.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20180225/fee2fb87/attachment.html>