RE Module Performance

[MRAB]

On 28/07/2013 19:13, wxjmfauth at gmail.com wrote:
> Le dimanche 28 juillet 2013 05:53:22 UTC+2, Ian a écrit :
>> On Sat, Jul 27, 2013 at 12:21 PM,  <wxjmfauth at gmail.com> wrote:
>>
>> > Back to utf. utfs are not only elements of a unique set of encoded
>>
>> > code points. They have an interesting feature. Each "utf chunk"
>>
>> > holds intrisically the character (in fact the code point) it is
>>
>> > supposed to represent. In utf-32, the obvious case, it is just
>>
>> > the code point. In utf-8, that's the first chunk which helps and
>>
>> > utf-16 is a mixed case (utf-8 / utf-32). In other words, in an
>>
>> > implementation using bytes, for any pointer position it is always
>>
>> > possible to find the corresponding encoded code point and from this
>>
>> > the corresponding character without any "programmed" information. See
>>
>> > my editor example, how to find the char under the caret? In fact,
>>
>> > a silly example, how can the caret can be positioned or moved, if
>>
>> > the underlying corresponding encoded code point can not be
>>
>> > dicerned!
>>
>>
>>
>> Yes, given a pointer location into a utf-8 or utf-16 string, it is
>>
>> easy to determine the identity of the code point at that location.
>>
>> But this is not often a useful operation, save for resynchronization
>>
>> in the case that the string data is corrupted.  The caret of an editor
>>
>> does not conceptually correspond to a pointer location, but to a
>>
>> character index.  Given a particular character index (e.g. 127504), an
>>
>> editor must be able to determine the identity and/or the memory
>>
>> location of the character at that index, and for UTF-8 and UTF-16
>>
>> without an auxiliary data structure that is a O(n) operation.
>>
>>
>>
>> > 2) Take a look at this. Get rid of the overhead.
>>
>> >
>>
>> >>>> sys.getsizeof('b'*1000000 + 'c')
>>
>> > 1000026
>>
>> >>>> sys.getsizeof('b'*1000000 + '€')
>>
>> > 2000040
>>
>> >
>>
>> > What does it mean? It means that Python has to
>>
>> > reencode a str every time it is necessary because
>>
>> > it works with multiple codings.
>>
>>
>>
>> Large strings in practical usage do not need to be resized like this
>>
>> often.  Python 3.3 has been in production use for months now, and you
>>
>> still have yet to produce any real-world application code that
>>
>> demonstrates a performance regression.  If there is no real-world
>>
>> regression, then there is no problem.
>>
>>
>>
>> > 3) Unicode compliance. We know retrospectively, latin-1,
>>
>> > is was a bad choice. Unusable for 17 European languages.
>>
>> > Believe of not. 20 years of Unicode of incubation is not
>>
>> > long enough to learn it. When discussing once with a French
>>
>> > Python core dev, one with commit access, he did not know one
>>
>> > can not use latin-1 for the French language!
>>
>>
>>
>> Probably because for many French strings, one can.  As far as I am
>>
>> aware, the only characters that are missing from Latin-1 are the Euro
>>
>> sign (an unfortunate victim of history), the ligature œ (I have no
>>
>> doubt that many users just type oe anyway), and the rare capital Ÿ
>>
>> (the miniscule version is present in Latin-1).  All French strings
>>
>> that are fortunate enough to be absent these characters can be
>>
>> represented in Latin-1 and so will have a 1-byte width in the FSR.
>
> ------
>
> latin-1? that's not even truth.
>
>>>> sys.getsizeof('a')
> 26
>>>> sys.getsizeof('ü')
> 38
>>>> sys.getsizeof('aa')
> 27
>>>> sys.getsizeof('aü')
> 39
>

 >>> sys.getsizeof('aa') - sys.getsizeof('a')
1

One byte per codepoint.

 >>> sys.getsizeof('üü') - sys.getsizeof('ü')
1

Also one byte per codepoint.

 >>> sys.getsizeof('ü') - sys.getsizeof('a')
12

Clearly there's more going on here.

FSR is an optimisation. You'll always be able to find some
circumstances where an optimisation makes things worse, but what
matters is the overall result.