[Tutor] Unicode trouble

[Kent Johnson]

Øyvind wrote:
>>The important question is, what is actual encoding of your source data?
>>
>>>Is there anything else I could try?
> 
> 
>>Understand why the above question is important, then answer it. Until you
> 
> do >you are just thrashing around in the dark.
> 
> The source is a text-document that as far as I know only contains English
> and Norwegian letters. It can be opened with Notepad and Excel. I tried to
> run thru it in Python by:
> 
> f = open('c://file.txt')
> 
> for i in f:
>     print f
> 
> and that doesn't seem to give any problem. It prints all characters
> without any trouble.

That doesn't narrow it down much though it does point towards latin-1 (or cp1252).

> How would I find what encoding the document is in? All I can find is by
> opening Notepad, selecting Font/Script and it says 'Western'.

That doesn't really mean anything about the doc. Try opening the file in your browser. Most browsers have an encoding menu (View / Character Encoding in Firefox, View / Encoding in IE). Find the selection in this menu that makes the text display correctly; that's the encoding of the file.

> Might the problem only be related to Win32com, not Python since Python
> prints it without trouble?

That's another issue. First you need to know what you are starting with.
> 
>>Do you know what a character encoding is? Do you understand the
 difference >between utf-8 and latin-1?
> 
> Earlier characters had values 1-255. (Ascii). Now, you have a wider
> choice. In our part of the world we can use an extended version which
> contains a lot more, latin-1. UTF-8 is a part of Unicode and contains a
> lot more characters than Ascii.
> 
> My knowledge about character encoding doesn't go much farther than this.
> Simply said, I understand that the document that I want to read includes
> characters beyond Ascii, and therefore I need to use UTF-8 or Latin-1. Why
> I should use one instead of the other, I have no idea.

You really should read this:
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
http://www.joelonsoftware.com/articles/Unicode.html

Kent
-- 
http://www.kentsjohnson.com