RE Module Performance

[Terry Reedy]

On 7/24/2013 2:15 PM, Chris Angelico wrote:
> On Thu, Jul 25, 2013 at 3:52 AM, Terry Reedy <tjreedy at udel.edu> wrote:

>> For my purpose, the mock Text works the same in 2.7 and 3.3+.
>
> Thanks for that report! And yes, it's going to behave exactly the same
> way, because its underlying structure is an ordered list of ordered
> lists of Unicode codepoints, ergo 3.3/PEP 393 is merely a question of
> performance. But if you put your code onto a narrow build, you'll have
> issues as seen below.

I carefully said 'For my purpose', which is to replace the tk Text 
widget. Up to 8.5, Tk's text is something like Python's narrow-build 
unicode.

If put astral chars into the toy editor, then yes, it would not work on 
narrow builds, but would on 3.3+.

  ...

 > If nobody had ever thought of doing a multi-format string
> representation, I could well imagine the Python core devs debating
> whether the cost of UTF-32 strings is worth the correctness and
> consistency improvements... and most likely concluding that narrow
> builds get abolished. And if any other language (eg ECMAScript)
> decides to move from UTF-16 to UTF-32, I would wholeheartedly support
> the move, even if it broke code to do so.

Making a UTF-16 implementation correct requires converting abstract 
'character' array indexes to concrete double byte array indexes. The 
simple O(n) method of scanning the string from the beginning for each 
index operation is too slow. When PEP393 was being discussed, I devised 
a much faster way to do the conversion.

The key idea is to add an auxiliary array of the abstract indexes of the 
astral chars in the abstract array. This is easily created when the 
string is created and can be done afterward with one linear scan (which 
is how I experimented with Python code). The length of that array is the 
number of surrogate pairs in the concrete 16-bit codepoint array. 
Subtracting that number from the length of the concrete array gives the 
length of the abstract array.

Given a target index of a character in the abstract array, use the 
auxiliary array to determine k, the number of astral characters that 
precede the target character. That can be done with either a O(k) linear 
scan or O(log k) binary search. Add 2 * k to the abstract index to get 
the corresponding index in the concrete array. When slicing a string 
with i0 and i1, slice the auxiliary array with k0 and k1 and adjusting 
the contained indexes downward to get the corresponding auxiliary array.

> To my mind, exposing UTF-16 surrogates to the application is a bug
 > to be fixed, not a feature to be maintained.

It is definitely not a feature, but a proper UTF-16 implementation would 
not expose them except to codecs, just as with the PEP 393 
implementation. (In both cases, I am excluding the sys size function as 
'exposing to the application'.)

 > But since we can get the best of both worlds with only
> a small amount of overhead, I really don't see why anyone should be
> objecting.

I presume you are referring to the PEP 393 1-2-4 byte implementation. 
Given how well it has been optimized, I think it was the right choice 
for Python. But a language that now uses USC2 or defective UTF-16 on all 
platforms might find the auxiliary array an easier fix.

-- 
Terry Jan Reedy