Python 3.0b2 cannot map '\u12b'

[Marc 'BlackJack' Rintsch]

On Mon, 01 Sep 2008 14:25:01 -0400, Terry Reedy wrote:

> Marc 'BlackJack' Rintsch wrote:
>> On Mon, 01 Sep 2008 02:27:54 -0400, Terry Reedy wrote:
>>
>> Most likely because Python figured out that the terminal expects cp437.
>> What does `sys.stdout.encoding` say?
> 
> The interpreter in the command prompt window says CP437. The IDLE Window
> says 'cp1252', and it handles the character fine. Given that Windows OS
> can handle the character, why is Python/Command Prompt limiting output?

The windows command prompt expects cp437 because that's what old DOS 
programs print to it.

> Characters the IDLE window cannot display (like surrogate pairs) it
> displays as boxes.  But if I cut '[][]' (4 chars) and paste into
> Firefox, I get 3 chars. '[]' where [] has some digits instead of being
> empty.  It is really confusing when every window on 'unicode-based'
> Windows handles a different subset.

That's because it is not 'unicode-based'.  Communication between those 
programs has to be done with bytes, so the sender has to encode unicode 
characters in the encoding the receiver expects.

> Is this the fault of Windows or of Python and IDLE (those two being
> more limited that FireFox)?

It's nobodies fault.  That's simply how the encoding stuff works.
 
>>> To put it another way, how can one 'choose' utf-8 for display to
>>> screen?
>> 
>> If the terminal expects cp437 then displaying utf-8 might give some
>> problems.
> 
> My screen displays whatever Windows tells the graphics card to tell the
> screen to display.

But the terminal gets bytes and expects them to be cp437 encoded 
characters and not utf-8.  So you can't send whatever unicode character 
you want, at least not without changing the encoding of the terminal.

> In OpenOffice, I can select a unicode font that displays at least
> everything in the BasicMultilingualPlane (BMP).

But OOo works with unicode internally, so there's no communication with 
outside programs involved here.

Ciao,
	Marc 'BlackJack' Rintsch