[issue14579] CVE-2012-2135: Vulnerability in the utf-16 decoder after error handling
Antoine Pitrou
report at bugs.python.org
Thu Apr 26 13:54:13 CEST 2012
Antoine Pitrou <pitrou at free.fr> added the comment:
> I ran tests of utf16_error_handling-3.2_4.patch on Python 3.1. Two tests are failing:
> - b'\x00\xd8'.decode('utf-16le', 'replace')='\ufffd\ufffd' != '\ufffd'
> - b'\xd8\x00'.decode('utf-16be', 'replace')='\ufffd\ufffd' != '\ufffd'
>
> I don't think that the test is correct: UTF-16 should resynchronize as
> early as possible (ignore the first invalid byte and restart at the
> following byte), so '\ufffd\ufffd' is the correct answer.
UTF-16 units are 16-bit words, not bytes, so '\uffffd' sounds correct to
me. You resynchronize on the word boundary: the invalid word is skipped.
> - with UTF-8 decoder: (b'\xC3' +
> '\xe9'.encode('utf-8')).decode('utf-8', 'replace') returns '\ufffd
> \xe9'
That's because UTF-8 operates on bytes: the invalid byte is skipped.
----------
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue14579>
_______________________________________
More information about the Python-bugs-list
mailing list