[Python-Dev] Unicode literals in Python 2.7

[Adam Bartoš]

I am in Windows and my terminal isn't utf-8 at the beginning, but I install
custom sys.std* objects at runtime and I also install custom readline hook,
so the interactive loop gets the input from my stream objects via
PyOS_Readline. So when I enter u'α', the tokenizer gets b"u'\xce\xb1'",
which is the string encoded in utf-8, and sys.stdin.encoding == 'utf-8'.
However, the input is then interpreted as u'\xce\xb1' instead of u'\u03b1'.

On Wed, Apr 29, 2015 at 6:40 PM, Guido van Rossum <guido at python.org> wrote:

> I suspect the interactive session is *not* always in UTF8. It probably
> depends on the keyboard mapping of your terminal emulator. I imagine in
> Windows it's the current code page.
>
> On Wed, Apr 29, 2015 at 9:19 AM, Adam Bartoš <drekin at gmail.com> wrote:
>
>> Yes, that works for eval. But I want it for code entered during an
>> interactive session.
>>
>> >>> u'α'
>> u'\xce\xb1'
>>
>> The tokenizer gets b"u'\xce\xb1'" by calling PyOS_Readline and it knows
>> it's utf-8 encoded. But the result of evaluation is u'\xce\xb1'. Because of
>> how eval works, I believe that it would work correctly if the
>> PyCF_SOURCE_IS_UTF8 was set, but it is not. That is why I'm asking if there
>> is a way to set it. Also, my naive thought is that it should be always set
>> in the case of interactive session.
>>
>>
>> On Wed, Apr 29, 2015 at 4:59 PM, Victor Stinner <victor.stinner at gmail.com
>> > wrote:
>>
>>> Le 29 avr. 2015 10:36, "Adam Bartoš" <drekin at gmail.com> a écrit :
>>> > Why I'm talking about PyCF_SOURCE_IS_UTF8? eval(u"u'\u03b1'") ->
>>> u'\u03b1' but eval(u"u'\u03b1'".encode('utf-8')) -> u'\xce\xb1'.
>>>
>>> There is a simple option to get this flag: call eval() with unicode, not
>>> with encoded bytes.
>>>
>>> Victor
>>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150429/132c7693/attachment.html>