[Python-3000] Unicode and OS strings

[Guido van Rossum]

On 9/18/07, Jim Jewett <jimjjewett at gmail.com> wrote:
> On 9/18/07, Stephen J. Turnbull <stephen at xemacs.org> wrote:
>
> > There's no UTF-8 in Python's internal string encoding.  What are you
> > talking about?
>
> (At least as of a few days ago)
>
> In Python 3 there is; strings are unicode.  A PyUnicodeObject object
> has two encodings that you can grab from a pointer (which means they
> have to be there; you don't have time to generate them like you would
> with a function pointer).

Incorrect. The pointer can be NULL. The API for getting the UTF-8
encoding is a function (moreover a function whose name starts with
_Py).

> One of these (str) is the "internal encoding" which is chosen at
> compile time, and the other (defenc) is now hard-coded to UTF-8.
>
> Hashing is also based on the UTF-8 bytestring.

Not any more as of a few hours ago; the hashing based on UTF-8 was
excessively expensive, and I rewrote it to directly use the code
units(?) (or whatever they are called -- the Py_UNICODE values). For
strings not using code units(?) > 2**16 this will give the same value
on all platforms; if there are code units(?) >= 2**16 results vary
since these will be represented as surrogates on 2-byte systems but
not on 4-byte systems.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)