unicode

[Erik Max Francis]

7stud wrote:

> Based on this example and the error:
> 
> -----
> u_str = u"abc\u9999"
> print u_str
> 
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u9999' in
> position 3: ordinal not in range(128)
> ------
> 
> it looks like when I try to display the string, the ascii decoder
> parses each character in the string and fails when it can't convert a
> numerical code that is higher than 127 to a character, i.e. the
> character \u9999.

If you try to print a Unicode string, then Python will attempt to first 
encode it using the default encoding for that file.  Here, it's apparent 
the default encoding is 'ascii', so it attempts to encode it into ASCII, 
which it can't do, hence the exception.  The error is no different from 
this:

 >>> u_str = u'abc\u9999'
 >>> u_str.encode('ascii')
Traceback (most recent call last):
   File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\u9999' in 
position 3: ordinal not in range(128)

> In the following example, I use encode() to convert a unicode string
> to a regular string:
> 
> -----
> u_str = u"abc\u9999"
> reg_str = u_str.encode("utf-8")
> print repr(reg_str)
> -----
> 
> and the output is:
> 
> 'abc\xe9\xa6\x99'
> 
> 1) Why aren't the characters 'a', 'b', and 'c' in hex notation?  It
> looks like python must be using the ascii decoder to parse the
> characters in the string again--with the result being python converts
> only the 1 byte numerical codes to characters. 2) Why didn't that
> cause an error like above for the 3 byte character?

Since you've already encoded the Unicode object as a normal string, 
Python isn't trying to do any implicit encoding.  As for why 'abc' 
appears in plain text, that's just the way repr works:

 >>> s = 'a'
 >>> print repr(s)
'a'
 >>> t = '\x99'
 >>> print repr(t)
'\x99'

repr is attempting to show the string in the most readable fashion.  If 
the character is printable, then it just shows it as itself.  If it's 
unprintable, then it shows it in hex string escape notation.

> Then if I try this:
> 
> ---
> u_str = u"abc\u9999"
> reg_str = u_str.encode("utf-8")
> print reg_str
> ---
> 
> I get the output:
> 
> abc<some chinese character>
> 
> Here it looks like python isn't using the ascii decoder anymore.  2)
> What determines which decoder python uses?

Again, that's because by already encoding it as a string, Python isn't 
doing any implicit encoding.  So it prints the raw string, which happens 
to be UTF-8, and which your terminal obviously supports, so you see the 
proper character.

-- 
Erik Max Francis && max at alcyone.com && http://www.alcyone.com/max/
  San Jose, CA, USA && 37 20 N 121 53 W && AIM, Y!M erikmaxfrancis
   Let us not seek the Republican answer or the Democratic answer but
    the right answer. -- John F. Kennedy