[issue12892] UTF-16 and UTF-32 codecs should reject (lone) surrogates
Serhiy Storchaka
report at bugs.python.org
Mon Sep 2 17:34:08 CEST 2013
Serhiy Storchaka added the comment:
Here is a patch which combines both Kang-Hao's patches, synchronized with tip, fixed and optimized.
Unfortunately even optimized this patch slowdown encoding/decoding some data. Here are some benchmark results (benchmarking tools are here: https://bitbucket.org/storchaka/cpython-stuff/src/default/bench).
3.3 3.4 3.4
unpatched patched
969 (+12%) 978 (+11%) 1087 encode utf-16le 'A'*10000
2453 (-62%) 2356 (-61%) 923 encode utf-16le '\u0100'*10000
222 (+12%) 224 (+11%) 249 encode utf-16le '\U00010000'+'\u0100'*9999
784 (+6%) 791 (+5%) 831 encode utf-16be 'A'*10000
750 (-4%) 752 (-4%) 719 encode utf-16be '\u0100'*10000
233 (+2%) 235 (+1%) 238 encode utf-16be '\U00010000'+'\u0100'*9999
531 (-7%) 545 (-9%) 494 encode utf-32le 'A'*10000
383 (-38%) 385 (-38%) 239 encode utf-32le '\u0100'*10000
324 (-24%) 325 (-25%) 245 encode utf-32le '\U00010000'+'\u0100'*9999
544 (-10%) 545 (-10%) 492 encode utf-32be 'A'*10000
384 (-38%) 384 (-38%) 239 encode utf-32be '\u0100'*10000
325 (-25%) 325 (-25%) 245 encode utf-32be '\U00010000'+'\u0100'*9999
682 (+5%) 679 (+5%) 715 decode utf-16le 'A'*10000
607 (+1%) 610 (+1%) 614 decode utf-16le '\u0100'*10000
550 (+1%) 554 (+0%) 556 decode utf-16le '\U00010000'+'\u0100'*9999
609 (+0%) 600 (+2%) 610 decode utf-16be 'A'*10000
464 (+1%) 466 (+1%) 470 decode utf-16be '\u0100'*10000
432 (+1%) 431 (+1%) 435 decode utf-16be '\U00010000'+'\u0100'*9999
103 (+272%) 374 (+2%) 383 decode utf-32le 'A'*10000
91 (+264%) 390 (-15%) 331 decode utf-32le '\u0100'*10000
90 (+257%) 393 (-18%) 321 decode utf-32le '\U00010000'+'\u0100'*9999
103 (+269%) 393 (-3%) 380 decode utf-32be 'A'*10000
91 (+263%) 406 (-19%) 330 decode utf-32be '\u0100'*10000
90 (+257%) 393 (-18%) 321 decode utf-32be '\U00010000'+'\u0100'*9999
Performance of utf-16 decoding is not changed. utf-16 encoder is 2.5 times slowed for UCS2 data (it was just memcpy) but still 3+ times faster than 2.7 and 3.2 (issue15026). Due to additional optimization it now even slightly faster for some other data. There is a patch for speed up UTF-32 encoding (issue15027), it should help to compensate it's performance degradation. UTF-32 decoder already 3-4 times faster than in 3.3 (issue14625).
I don't see performance objection against this patch.
----------
Added file: http://bugs.python.org/file31555/utf_16_32_surrogates_2.patch
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12892>
_______________________________________
More information about the Python-bugs-list
mailing list