[Patches] [ python-Patches-1333679 ] Allow use of non-latin1 chars in interactive shell

[SourceForge.net]

Patches item #1333679, was opened at 2005-10-21 02:49
Message generated for change (Comment added) made by loewis
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1333679&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: IDLE
Group: None
Status: Closed
Resolution: Rejected
Priority: 5
Submitted By: Noam Raphael (noamr)
Assigned to: Nobody/Anonymous (nobody)
Summary: Allow use of non-latin1 chars in interactive shell

Initial Comment:
If I type a unicode string, such as u"שלום" in the
interactive interpreter in the terminal, it does what I
expect - return the unicode string (in this case,
u'\u05e9\u05dc\u05d5\u05dd')

However, if I type it in IDLE's interactive
interpreter, I get something funny - I get a unicode
string (u'\xd7\xa9\xd7\x9c\xd7\x95\xd7\x9d'), which is
actually u"שלום".decode('utf8').encode('latin1').

This is caused by the PyShell.runsource method, which
checks if the source string it gets is unicode, and if
so, encodes it using the 'ascii' codec. If this check
is removed, so that the unicode object itself gets
compiled, everything works out fine.

The bottom line: remove the if block starting with "if
isinstance(source, types.UnicodeType)", at
PyShell.py:589, and everything is fine.

Have a good day,
Noam

----------------------------------------------------------------------

>Comment By: Martin v. Löwis (loewis)
Date: 2005-11-02 19:13

Message:
Logged In: YES 
user_id=21627

No, IOBindings.encoding is not hard coded 'latin1'. It is
locale.getdefaultlocale()[1] on Windows, and
locale.nl_langinfo(locale.CODESET) on Unix. If these fail,
it is 'ascii'.

----------------------------------------------------------------------

Comment By: Noam Raphael (noamr)
Date: 2005-11-02 12:27

Message:
Logged In: YES 
user_id=679426

But IOBindings.encoding is a hard coded 'latin1', which is 
certainly not better than 'utf8'.

Is there a way to detect the encoding from the current locale?

Noam

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2005-11-02 05:43

Message:
Logged In: YES 
user_id=21627

This isn't right, though: it will assume that the source is
UTF-8 encoded, even though the user would expect a different
encoding.

The "right" way would be if IDLE compiled the source with
IOBindings.encoding.

Rejecting this patch.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1333679&group_id=5470