[IPython-dev] Bug: unicode and IPython

[Fernando Perez]

On 8/10/06, Alexander Belchenko <bialix at ukr.net> wrote:
> This bug appears in IPython with readline installed and without also. It
> was with in Gary Bishop UNC readline and is with pyreadline 1.3 too.
> When I type russian text as unicode string (u"") IPython does not decode
> it to unicode correctly. Standard python interpreter does it well.
>
> Below is reproduction for russian word "Тест" (it's means "Test" in english)
>
> Standard python interpreter:
>
> Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on
> win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> a = u"Тест"
> >>> a
> u'\u0422\u0435\u0441\u0442'
>
> All correct.
>
>
> IPython session:
>
> Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)]
> Type "copyright", "credits" or "license" for more information.
>
> IPython 0.7.2 -- An enhanced Interactive Python.
> ?       -> Introduction to IPython's features.
> %magic  -> Information about IPython's 'magic' % functions.
> help    -> Python's own help system.
> object? -> Details about 'object'. ?object also works, ?? prints more.
>
> In [1]: a = u"Тест"
>
> In [2]: a
> Out[2]: u'\x92\xa5\xe1\xe2'
>
> As you can see it's incorrect -- it's not russian unicode string.

Hmm, I know precious little about unicode, so I'm not too surprised to
hear of problems.  Here's what I see on my system:

planck[travel]> python
Python 2.3.4 (#1, Feb  2 2005, 12:11:53)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = u"Тест"
>>> a
u'\xd0\xa2\xd0\xb5\xd1\x81\xd1\x82'

Could you please try this?

>>> str(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode characters in position
0-7: ordinal not in range(128)
>>>

I actually get identical results for python and ipython, so it must be
a locale-related thing.  If you can tell us what environment variables
need to be set, perhaps we might track this one down.

Help from non-US-based users would be greatly appreciated, I'm a bit lost here.

Cheers,

f