Python usage numbers

[Dave Angel]

On 02/12/2012 05:27 PM, Roy Smith wrote:
> In article<mailman.5739.1329084873.27778.python-list at python.org>,
>   Chris Angelico<rosuav at gmail.com>  wrote:
>
>> On Mon, Feb 13, 2012 at 9:07 AM, Terry Reedy<tjreedy at udel.edu>  wrote:
>>> The situation before ascii is like where we ended up *before* unicode.
>>> Unicode aims to replace all those byte encoding and character sets with
>>> *one* byte encoding for *one* character set, which will be a great
>>> simplification. It is the idea of ascii applied on a global rather that
>>> local basis.
>> Unicode doesn't deal with byte encodings; UTF-8 is an encoding, but so
>> are UTF-16, UTF-32. and as many more as you could hope for. But
>> broadly yes, Unicode IS the solution.
> I could hope for one and only one, but I know I'm just going to be
> disapointed.  The last project I worked on used UTF-8 in most places,
> but also used some C and Java libraries which were only available for
> UTF-16.  So it was transcoding hell all over the place.
>
> Hopefully, we will eventually reach the point where storage is so cheap
> that nobody minds how inefficient UTF-32 is and we all just start using
> that.  Life will be a lot simpler then.  No more transcoding, a string
> will just as many bytes as it is characters, and everybody will be happy
> again.

Keep your in-memory character strings as Unicode, and only 
serialize(encode) them when they go to/from a device, or to/from 
anachronistic code.  Then the cost is realized at the point of the 
problem.  No different than when deciding how to serialize any other 
data type.  Do it only at the point of entry/exit of your program.

But as long as devices are addressed as bytes, or as anything smaller 
than 32bit thingies, you will have encoding issues when writing to the 
device, and decoding issues when reading.  At the very least, you have 
big-endian/little-endian ways to encode that UCS-4 code point.