hex dump w/ or w/out utf-8 chars

[wxjmfauth at gmail.com]

Le jeudi 11 juillet 2013 15:32:00 UTC+2, Chris Angelico a écrit :
> On Thu, Jul 11, 2013 at 11:18 PM,  <wxjmfauth at gmail.com> wrote:
> 
> > Just to stick with this funny character ẞ, a ucs-2 char
> 
> > in the Flexible String Representation nomenclature.
> 
> >
> 
> > It seems to me that, when one needs more than ten bytes
> 
> > to encode it,
> 
> >
> 
> >>>> sys.getsizeof('a')
> 
> > 26
> 
> >>>> sys.getsizeof('ẞ')
> 
> > 40
> 
> >
> 
> > this is far away from the perfection.
> 
> 
> 
> Better comparison is to see how much space is used by one copy of it,
> 
> and how much by two copies:
> 
> 
> 
> >>> sys.getsizeof('aa')-sys.getsizeof('a')
> 
> 1
> 
> >>> sys.getsizeof('ẞẞ')-sys.getsizeof('ẞ')
> 
> 2
> 
> 
> 
> String objects have overhead. Big deal.
> 
> 
> 
> > BTW, for a modern language, is not ucs2 considered
> 
> > as obsolete since many, many years?
> 
> 
> 
> Clearly. And similarly, the 16-bit integer has been completely
> 
> obsoleted, as there is no reason anyone should ever bother to use it.
> 
> Same with the float type - everyone uses double or better these days,
> 
> right?
> 
> 
> 
> http://www.postgresql.org/docs/current/static/datatype-numeric.html
> 
> http://www.cplusplus.com/doc/tutorial/variables/
> 
> 
> 
> Nope, nobody uses small integers any more, they're clearly completely obsolete.
> 
> 
> 

Sure there is some overhead because a str is a class.
It still remain that a "ẞ" weights 14 bytes more than
an "a".

In "aẞ", the ẞ weights 6 bytes.

>>> sys.getsizeof('a')
26
>>> sys.getsizeof('aẞ')
42

and in "aẞẞ", the ẞ weights 2 bytes

sys.getsizeof('aẞẞ')

And what to say about this "ucs4" char/string '\U0001d11e' which
is weighting 18 bytes more than an "a".

>>> sys.getsizeof('\U0001d11e')
44

A total absurdity. How does is come? Very simple, once you
split Unicode in subsets, not only you have to handle these
subsets, you have to create "markers" to differentiate them.
Not only, you produce "markers", you have to handle the
mess generated by these "markers". Hiding this markers
in the everhead of the class does not mean that they should
not be counted as part of the coding scheme. BTW, since
when a serious coding scheme need an extermal marker?



>>> sys.getsizeof('aa') - sys.getsizeof('a')
1

Shortly, if my algebra is still correct:

(overhead + marker + 2*'a') - (overhead + marker + 'a')
= (overhead + marker + 2*'a') - overhead - marker - 'a')
= overhead - overhead + marker - marker + 2*'a' - 'a'
= 0 + 0 + 'a'
= 1

The "marker" has magically disappeared.

jmf