[ python-Bugs-1704793 ] incorrect return value of unicodedata.lookup() - beoynd BMP

[SourceForge.net]

Bugs item #1704793, was opened at 2007-04-21 12:52
Message generated for change (Comment added) made by lemburg
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1704793&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Unicode
Group: Python 2.5
Status: Open
Resolution: None
Priority: 6
Private: No
Submitted By: vbr (vlbrom)
>Assigned to: Martin v. Löwis (loewis)
Summary: incorrect return value of unicodedata.lookup() - beoynd BMP

Initial Comment:
There seem to be an incorrect handling of unicode characters beyond the BMP (code point higher than 0xFFFF) in the unicodedata module - function lookup() on narrow unicode python builds (python 2.5.1, Windows XPh)

>>> unicodedata.lookup("GOTHIC LETTER FAIHU")
u'\u0346'
(should be u'\U00010346' - the beginning of the literal is truncated - leading to the ambiguity - in this case u'\u0346' is a combining diacritics "COMBINING BRIDGE ABOVE")

on the contrary, the unicode string literals \N{name} work well.

>>> u"\N{GOTHIC LETTER FAIHU}"
u'\U00010346'

Unfortunately, I haven't been able to find the problematic pieces of sourcecode, so I'm not able to fix it. 

It seems, that initially the correct information on the given codepoint is used, but finally only the last four digits of the codepoint value are taken into account using the "narrow" unicode literal \uxxxx instead of \Uxxxxxxxx 
, while the same task is handled correctly by the unicodeescape codec used for unicode string literals.

vbr



----------------------------------------------------------------------

>Comment By: M.-A. Lemburg (lemburg)
Date: 2007-06-12 11:35

Message:
Logged In: YES 
user_id=38388
Originator: NO

Martin, please have a look. Thanks.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2007-04-21 22:29

Message:
Logged In: YES 
user_id=849994
Originator: NO

Confirmed with an linux-x86 UCS-4 build here.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1704793&group_id=5470