[issue30717] Add unicode grapheme cluster break algorithm

[Steven D'Aprano]

Steven D'Aprano <steve+python at pearwood.info> added the comment:

> I think it would be a mistake to make the stdlib use this for most 
> notions of what a "character" is, as I said this notion is also 
> inaccurate. Having an iterator library somewhere that you can use and 
> compose is great, changing the internal workings of string operations 
> would be a major change, and not entirely productive.

Agreed. 

I won't pretend to be able to predict what Python 5.0 will bring *wink* 
but there's too much history around the "code point = character" notion 
for the language to change now.

If the language can expose a grapheme iterator, then people can 
experiment with grapheme-based APIs in libraries.

(By grapheme I mean "extended grapheme cluster", but that's a mouthful. 
Sorry linguists.)

What do you think of these as a set of grapheme primitives?

(1) is_grapheme_break(string, i)

Return True if a grapheme break would occur *before* string[i].

(2) graphemes(string, start=0, end=len(string))

Iterate over graphemes in string[start:end].

(3) graphemes_reversed(string, start=0, end=len(string))

Iterate over graphemes in reverse order.

I *think* is_grapheme_break would be enough for people to implement 
their own versions of graphemes and graphemes_reversed. Here's an 
untested version:

    def graphemes(string, start, end):
        cluster = []
        for i in range(start, end):
            c = string[i]
            if is_grapheme_break(string, i):
                if i != start:
                    # don't yield the empty cluster at Start Of Text
                    yield ''.join(cluster)
                cluster = [c]
            else:
                cluster.append(c)
        if cluster:
            yield ''.join(cluster)

Regarding is_grapheme_break, if I understand the note here:

https://www.unicode.org/reports/tr29/#Testing

one never needs to look at more than two adjacent code points to tell 
whether or not a grapheme break will occur between them, so this ought 
to be pretty efficient. At worst, it needs to look at string[i-1] and 
string[i], if they exist.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue30717>
_______________________________________