UnicodeEncodeError when not running script from IDE

[Dave Angel]

On 02/12/2013 12:12 PM, Magnus Pettersson wrote:
>> < snip >
>>
> #Here kanji = u"私"
> baseurl = u"http://www.romajidesu.com/kanji/"
> url = baseurl+kanji
> savefile([url]) #this test works now. uses: io.open(filepath, "a",encoding="UTF-8") as f:
> # This made the fetching of the website work.

You don't show the code that actually does the io.open(), nor the 
url.encode, so I'm not going to guess what you're actually doing.


> Why did i have to write url.encode("UTF-8") when url already is unicode? I feel i dont have a good understanding of this.
> page = urllib2.urlopen(url.encode("UTF-8"))

utf-8 is NOT unicode;  they are entirely different.  Unicode is 
conceptually 32 bits per character, and is an internal representation. 
There are a million or so characters defined.  Nearly always when you're 
talking to an external device, you need bytes.  Since you can't cram 32 
bits into 8, you have to encode it.  Examples of devices would be any 
file, or the console.  Notice that sometimes you can use unicode 
directly for certain functions.  For example, the Windows file name is 
composed of Unicode characters, so Windows has function calls that 
accept Unicode directly.  But back to 8 bits:

One encoding is called ASCII, which is simply the bottommost 7 bits. 
But of course it gets an error if there are any characters above 127.

Other encodings try to pick an 8 bit subset of the million possible 
characters.  Again, if you happen to have a character that's not in that 
subset, you'll get an error.

There are also other encodings which are hard to describe, but 
fortunately pretty rare these days.

Then there's utf-8, which uses a variable length  bunch of bytes for 
each character.  It's designed to use the ASCII encoding for characters 
which are below 128, but uses two or more bytes for all the other 
characters.  So it works out well when most characters happen to be ASCII.

Once encoded, a stream of bytes can only be successfully interpreted if 
you use the same decoding when processing them.



-- 
DaveA