[Python-Dev] PyUnicode_GetMax() and PyUnicode_FromOrdinal() Was: Breaking undocumented API

[Alexander Belopolsky]

On Tue, Nov 16, 2010 at 1:57 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> Alexander Belopolsky wrote:
>> On Tue, Nov 16, 2010 at 1:06 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> ..
>>> Now, we can't use a macro for [PyUnicode_GetMax()], since the information has
>>> to be available as callable in order to applications or extensions
>>> to use it (without recompile).
>>>
>>
>> .. but it *is* a macro resolving to either PyUnicodeUCS2_GetMax or
>> PyUnicodeUCS4_GetMax.
>
> That doesn't count :-) It's only a trick to prevent external code
> from using the wrong Unicode APIs.
>
> There still is a real function behind the renaming.
>
>> What is the scenario when may want to change
>> what PyUnicodeUCS?_GetMax return and have extensions pick up the
>> change without a recompile?
>
> If an extensions uses the stable ABI, it will want to know
> whether the interpreter was built for UCS2 or UCS4 (even if
> it doesn't use the Unicode APIs directly).
>
>> UCS2 case will certainly never change
>> since it is already 0xFFFF.  Is it possible that USC4 will be expanded
>> beyond 0x10FFFF?
>
> Well, the Unicode Consortium decided to not go beyond 0x10FFFF,
> but then you never know... when they started out on the quest,
> 16 bits appeared more than enough, but they found out relatively
> quickly that the Asian scripts had enough code points to easily
> fill that space.
>
> Once space is available, it tends to get used sooner or later :-)
>
>> Note that we can have both a macro and a function
>> version.  This is fairly standard practice in Python C-API.
>
> Sure, but what for ?

Note that PyUnicode_FromOrdinal()  is documented (in unicodeobject.h)
as follows without a reference to PyUnicode_GetMax():

"""
   Create a Unicode Object from the given Unicode code point ordinal.

   The ordinal must be in range(0x10000) on narrow Python builds
   (UCS2), and range(0x110000) on wide builds (UCS4). A ValueError is
   raised in case it is not.
"""

The actual implementation actually checks UCS4 range only.

    if (ordinal < 0 || ordinal > 0x10ffff) {
	PyErr_SetString(PyExc_ValueError,
                        "chr() arg not in range(0x110000)");
        return NULL;
    }

This actually looks like a bug:

>>> len(chr(0x10FFFF))
2

(on a USC2 build.)

Also, I think PyUnicode_FromOrdinal()  should take Py_UNICODE argument
rather than int.