[New-bugs-announce] [issue5108] Invalid UTF-8 ("%s") length in PyUnicode_FromFormatV()

[STINNER Victor]

New submission from STINNER Victor <victor.stinner at haypocalc.com>:

PyUnicode_FromFormatV() doesn't count correctly the unicode length of 
an UTF-8 string. Commit r57837 "Change %s argument for 
PyUnicode_FromFormat to be UTF-8. Fixes #1070." introduced the bug. To 
compute the length, it uses a a complex code to compute the length of 
the UTF-8 string, whereas PyUnicode_DecodeUTF8(p, 
strlen(p), "replace") + Py_UNICODE_COPY() is used to copy the string. 
The problem may comes from the error handling ("replace").

Valgrind show that the error occurs at Py_UNICODE_COPY(): Invalid 
write of size 1. Since it's only one byte, Python does not always 
crash.

To reproduce the crash, use PyUnicode_FromFormatV() or function using 
it: PyUnicode_FromFormat(), PyErr_Format(), ...

Example 1:

    import grp
    
x=["\uDBE7\u8C99", "\u9C31\uF8DC\u3EC5\u1804\u629D\uE748\u68C8\uCF74\u9E63\uF647\uBF7A\uED63"]
    x=str(x)
    grp.getgrnam(x)

Example 2:

    import unicodedata
    x 
= "\\udbe7\u8c99', '\u9c31\\uf8dc\u3ec5\u1804\u629d\\ue748\u68c8\ucf74\u9e63\\uf647\ubf7a\\ued63"
    unicodedata.lookup(x)

I wrote a patch reusing PyUnicode_DecodeUTF8(p, strlen(p), "replace") 
+ PyUnicode_GET_SIZE() to get the real length of the converted UTF-8 
string.

A better patch should reuse code used to convert UTF-8 to Unicode with 
the "replace" error handling.

----------
components: Unicode
messages: 80814
nosy: haypo
severity: normal
status: open
title: Invalid UTF-8 ("%s") length in PyUnicode_FromFormatV()
versions: Python 3.0, Python 3.1

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5108>
_______________________________________