[issue8670] c_types.c_wchar should not assume that sizeof(wchar_t) == sizeof(Py_UNICODE)

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Patch for Python3:
 - Fix PyUnicode_AsWideChar() to support surrogates (Py_UNICODE: 2 bytes, wchar_t: 4 bytes)
 - u_set() of _ctypes uses PyUnicode_AsWideChar()
 - add a test (skipped if sizeof(wchar_t) is smaller than 4 bytes)

It's too late to fix Python2: 2.7 beta 2 was released (it doesn't support non BMP characters for chr()/ord()).

TODO:
 - I'm not sure that my patch on PyUnicode_AsWideChar() works if sizeof(wchar_t)==2
 - Test the patch on Windows
 - Check that it doesn't break other functions calling PyUnicode_AsWideChar()

ctypes, _locale.strcoll() and time.strftime() use PyUnicode_AsWideChar(). time has interesting comments:

    /* This assumes that PyUnicode_AsWideChar doesn't do any UTF-16
       expansion. */
    if (PyUnicode_AsWideChar((PyUnicodeObject*)format,
                             (wchar_t*)PyBytes_AS_STRING(tmpfmt),
                             PyUnicode_GetSize(format)+1) == (size_t)-1)
        /* This shouldn't fail. */
        Py_FatalError("PyUnicode_AsWideChar failed");

----------
keywords: +patch
Added file: http://bugs.python.org/file17322/pyunicode_aswidechar_surrogates-py3k.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue8670>
_______________________________________