Chardet, file, ... and the Flexible String Representation

[Ned Batchelder]

On 9/9/13 10:28 AM, wxjmfauth at gmail.com wrote:
> Le vendredi 6 septembre 2013 17:46:14 UTC+2, Piet van Oostrum a écrit :
>> wxjmfauth at gmail.com writes:
>>
>>
>>
>>> The Flexible String Representation has conceptually to
>>> face the same problem. It splits "unicode" in chunks and
>>> it has to solve two problems at the same time, the coding
>>> and the handling of multiple "char sets". The problem?
>>> It fails.
>>> "This poor Flexible String Representation does not succeed
>>> to solve the problem it create itsself."
>>
>>
>> The FSR does not split unicode in chuncks. It does not create problems and therefore it doesn't have to solve this.
>>
>>
>>
>> The FSR simply stores a Unicode string as an array[*] of ints (the Unicode code points of the characters of the string. That's it. Then it uses a memory-efficient way to store this array of ints. But that has nothing to do with character sets. The same principle could be used for any array of ints.
>>
>>
>>
>> So you are seeking problems where there are none. And you would have a lot more peace of mind if you stopped doing this.
>>
>>
>>
>> [*] array in the C sense.
>>
>> -- 
>>
>> Piet van Oostrum <piet at vanoostrum.org>
>>
>> WWW: http://pietvanoostrum.com/
>>
>> PGP key: [8DAE142BE17999C4]
> ----------
>
>
> Due to its nature, a character cann't be handled in the
> same way a one another type. That's the purpose of the UTF.
>
> -----
>
> Chunk latin-1, perfomance
>
> ref:
>>>> timeit.timeit("a = 'hundred'; 'x' in a")
> 0.13144639994075646
>
>>>> timeit.timeit("a = 'hundrez'; 'x' in a")
> 0.13780295544393084
>
> Chunk ucs2, perfomance
>
>>>> timeit.timeit("a = 'hundre€'; 'x' in a")
> 0.23505392241617074
>
> Chunk ucs4, perfomance
>
>>>> timeit.timeit("a = 'hundre\U0001d11e'; 'x' in a")
> 0.26266673650735584
>
> Comment: Such differences never happen with utf.
>
> -----
>
> Chunk latin-1, memory
>
>>>> sys.getsizeof('a')
> 26
>
> Chunk ucs2, memory
>
>>>> sys.getsizeof('€')
> 40
>
> Comment: 14 bytes more than latin-1
>
> Chunk ucs4, memory
>
>>>> sys.getsizeof('\U0001d11e')
> 44
>
> Comment: 18 bytes more than latin-1
>
> Comment: With utf, a char (in string or not) never exceed 4
>
> bytes.
>
> -----
>
> 'a' + '€' in utf, conceptually
>
> Concatenate the *unicode tranformation units*.
> Some kind of a real direct 'a' + '€'.
>
>
> 'a' + '€' in FSR, conceptually
>
> 1) Check the "internal coding" of 'a'
> 2) Check the "internal coding" of '€'
> 3) Compare these codings
>
> 4a) If they match, concatenate the bytes
>
> 4b) If they do not match
> 	5) Reencode the string which has to
> 	6) Concatenate
> 	7) Set the "internal coding" status for
> 	further processing
>
> -----
>
> Complicate and full of side effects, eg :
>
>>>> sys.getsizeof('a')
> 26
>>>> sys.getsizeof('aé')
> 39
>
> Is not a latin-1 "é" supposed to count as a latin-1 "a" ?
>
> ----
>
> I picked up random methods, there may be variations, basically
> this general behaviour is always expected.
>
>
> jmf
>

jmf, thanks for your reply.  You've calmed my fears that there is 
something wrong with the Flexible String Representation.  None of the 
examples you show demonstrate any behavior contrary to the Unicode spec.

--Ned.