Getting the encoding of sys.stdout and sys.stdin, and changing it properly

["Martin v. Löwis"]

velle at velle.dk wrote:
> Exactly what does the following line return?

On your system, the result of nl_langinfo(CODESET),
see nl_langinfo(3).

On your system, this, in turn, is computed from the LANG,
LC_CHARSET, and LC_ALL environment variables.

> Is it the encoding of the terminal? I think not, because when I change
> the encoding in my terminal the result is still the same.

It should be the encoding of your terminal. If you change the encoding
of your terminal, you should simultaneously also change your locale.

There is no way to determine the encoding of the terminal
programmatically, so the convention (on Unix) is that the locale setting
tells you what the terminal encoding is.

Unfortunately, there is also no way for the terminal to influence the
locale when it changes the encoding. So just don't do that, or do it
right.

> Is it the encoding of the string python "hands over" to the terminal?

Not sure what you mean by "hands over". There is no way for the program
to tell the terminal what encoding to use (*), so Python cannot hand
something over. If you really mean "the encoding of the string sent
to the terminal", then ... perhaps. This encoding will only be used
when sending Unicode strings, not when sending byte strings (of those,
Python cannot know what encoding they have, and it sends them
unmodified).

>>>>sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
>>>>sys.stdout.encoding
> 
> 'ISO-8859-1'
> 
> Then what?

Then you have two bugs. One bug is in Python: it never occurred
to me to set the .encoding attribute on a stream writer, but that
might be a good idea. You also have a bug in your code: you
shouldn't set sys.stdout to a UTF-8 writer if the underlying
stream assumes ISO-8859-1.

> 3)
> Does raw_input() come from sys.stdin?

Yes, it does.

> 4)
> The following script is not working, can you please tell me how to do
> it right.

Not exactly sure what you did (so it is hard to tell what to change),
however, I guess you should set your terminal to UTF-8 mode
(or, better, run the terminal in UTF-8 mode to begin with).

Alternatively, you should use the encoding that the terminal uses.

> write this unicode letter, Turkish che, unicode 0x00E7  ç
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/usr/lib/python2.3/codecs.py", line 295, in readline
>     return self.decode(line, self.errors)[0]
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-1:
> unexpected end of data
> 
> When prompted, I simply enter the che with my Turkish keyboard layout.

What do you get when you do

% hexdump -C
ç<enter><ctl-d>

I expect you see

00000000  e7 0a                                             |ç.|
00000002

which would indicate that your terminal sends latin-1.

Regards,
Martin

(*) Well, there is ISO 2022, but it doesn't really work, atleast not
in most terminals, and Python doesn't know how to use it, either.