[Cryptography-dev] hash.SHA256 cpu expensive per byte versus byte-string?

[lvh]

Hi Frank,

> On Jul 14, 2016, at 12:23 AM, Frank Siebenlist <frank.siebenlist at gmail.com> wrote:
> 
> Python's native hashing module (hashlib), shows similar results:
> - about the same time when passed the 8MB blob in one go
> (probably expected as both use openssl)
> - substantial overhead when looping over small chunks (up to 100 times)
> - except that it's about 6 times faster per single byte..

The perf by chunk is a consequence of how SHA256 works. The higher perf for many calls is a consequence of extension modules vs cffi.

lvh

> n: 8388608
> b:  1.958238124847412
> b2:  1.0818939208984375
> b8:  0.2987058162689209
> b32:  0.10640311241149902
> b64:  0.06242084503173828
> b128:  0.04123806953430176
> b256:  0.03258681297302246
> 8388608  bs:  0.02389383316040039
> 
> Guess hashlib used some better optimization on the C-calls (?).
> 
> This is my last update on this observation.
> Conclusion is "so be it", and using bigger chunks for hashing gives
> (much) better performance.
> 
> -Frank.
> 
> On Tue, Jul 12, 2016 at 10:49 AM, Frank Siebenlist
> <frank.siebenlist at gmail.com> wrote:
>> After I sent my message yesterday evening, I was also wondering about
>> that 512bit (64byte) block-size of sha256, and if that would add to
>> the observed slowness.
>> The following output shows time as a function of byte-chunk size
>> (1,2,8,32,64,128,256 bytes)
>> 
>> b:  12.111763954162598
>> b2:  5.806451082229614
>> b8:  1.4664850234985352
>> b32:  0.37551307678222656
>> b64:  0.20229697227478027
>> b128:  0.11141395568847656
>> b256:  0.06758689880371094
>> 8388608  bs:  0.020879030227661133
>> 
>> Time seems to go down linearly with increase of chunk size, and there
>> is no perceived "speed boost"  when we go through the 64byte
>> thresh-hold.
>> Time seems to be only linearly related to the number of python-to-C calls.
>> 
>> And again, I can understand that the overhead is proportional to the
>> number of python-to-C calls, but it's just the factor of 500 (2-3
>> order of magnitude) that (unpleasantly) surprised me. It requires one
>> to optimize on byte-string size to pass in the update(), when you have
>> many bytes to hash. For example, if you read from a file or socket,
>> don't update() 1 byte at the time while you read from the stream, but
>> fill-up a (big) buffer first and pass that buffer.
>> 
>> -Frank.
>> 
>> PS. I haven't looked at the sha256 C-code, but I can imagine that when
>> you pass the update() one byte at the time, it will fill-up some
>> 64byte-buffer, and if that buffer is filled, it will churn/hash that
>> block. The adding a byte to the buffer is all low-level fast code in
>> C, while the churning would use significantly more CPU cycles... hard
>> to phantom that you would see much slower performance when you pass a
>> single byte at the time in C...
>> 
>> 
>> On Tue, Jul 12, 2016 at 8:07 AM, lvh <_ at lvh.io> wrote:
>>> Hi,
>>> 
>>>> On Jul 11, 2016, at 10:42 PM, Frank Siebenlist <frank.siebenlist at gmail.com> wrote:
>>> 
>>> <snipsnip>
>>> 
>>>> I understand that there may be a few more object-creations and casts involved in the looping, but 500 times slower… that was un unexpected surprise.
>>> 
>>> As expected. You both get massively increased C call overhead and the worst case because you don’t get to hit a block until every 512/8 == 64 updates. Alas, openssl speed doesn’t distinguish between the same message sizes but in different chunk sizes, but you can at least clearly see the performance multiplier for larger messages.
>>> 
>>> lvh
>>> _______________________________________________
>>> Cryptography-dev mailing list
>>> Cryptography-dev at python.org
>>> https://mail.python.org/mailman/listinfo/cryptography-dev
> _______________________________________________
> Cryptography-dev mailing list
> Cryptography-dev at python.org
> https://mail.python.org/mailman/listinfo/cryptography-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 643 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/cryptography-dev/attachments/20160714/41869cf3/attachment.sig>