Chardet, file, ... and the Flexible String Representation

[wxjmfauth at gmail.com]

Le vendredi 6 septembre 2013 17:46:14 UTC+2, Piet van Oostrum a écrit :
> wxjmfauth at gmail.com writes:
> 
> 
> 
> > The Flexible String Representation has conceptually to
> 
> > face the same problem. It splits "unicode" in chunks and
> 
> > it has to solve two problems at the same time, the coding
> 
> > and the handling of multiple "char sets". The problem?
> 
> > It fails.
> 
> > "This poor Flexible String Representation does not succeed
> 
> > to solve the problem it create itsself."
> 
> 
> 
> The FSR does not split unicode in chuncks. It does not create problems and therefore it doesn't have to solve this. 
> 
> 
> 
> The FSR simply stores a Unicode string as an array[*] of ints (the Unicode code points of the characters of the string. That's it. Then it uses a memory-efficient way to store this array of ints. But that has nothing to do with character sets. The same principle could be used for any array of ints.
> 
> 
> 
> So you are seeking problems where there are none. And you would have a lot more peace of mind if you stopped doing this.
> 
> 
> 
> [*] array in the C sense.
> 
> -- 
> 
> Piet van Oostrum <piet at vanoostrum.org>
> 
> WWW: http://pietvanoostrum.com/
> 
> PGP key: [8DAE142BE17999C4]

----------


Due to its nature, a character cann't be handled in the
same way a one another type. That's the purpose of the UTF.

-----

Chunk latin-1, perfomance

ref:
>>> timeit.timeit("a = 'hundred'; 'x' in a")
0.13144639994075646

>>> timeit.timeit("a = 'hundrez'; 'x' in a")
0.13780295544393084

Chunk ucs2, perfomance

>>> timeit.timeit("a = 'hundre€'; 'x' in a")
0.23505392241617074

Chunk ucs4, perfomance

>>> timeit.timeit("a = 'hundre\U0001d11e'; 'x' in a")
0.26266673650735584

Comment: Such differences never happen with utf.

-----

Chunk latin-1, memory

>>> sys.getsizeof('a')
26

Chunk ucs2, memory

>>> sys.getsizeof('€')
40

Comment: 14 bytes more than latin-1

Chunk ucs4, memory

>>> sys.getsizeof('\U0001d11e')
44

Comment: 18 bytes more than latin-1

Comment: With utf, a char (in string or not) never exceed 4 

bytes.

-----

'a' + '€' in utf, conceptually

Concatenate the *unicode tranformation units*.
Some kind of a real direct 'a' + '€'.


'a' + '€' in FSR, conceptually

1) Check the "internal coding" of 'a'
2) Check the "internal coding" of '€'
3) Compare these codings

4a) If they match, concatenate the bytes

4b) If they do not match
	5) Reencode the string which has to
	6) Concatenate
	7) Set the "internal coding" status for
	further processing

-----

Complicate and full of side effects, eg :

>>> sys.getsizeof('a')
26
>>> sys.getsizeof('aé')
39

Is not a latin-1 "é" supposed to count as a latin-1 "a" ?

----

I picked up random methods, there may be variations, basically
this general behaviour is always expected.


jmf