UnicodeEncodeError when not running script from IDE

[Peter Otten]

Magnus Pettersson wrote:

> I have tried now to take away printing to terminal and just keeping the
> writing to a .txt file to disk (which is what the scripts purpose is):
> 
> with open(filepath,"a") as f:
>     for card in cardlist:
>         f.write(card+"\n")
> 
> The file it writes to exists and im just appending to it, but when i run
> the script trough eclipse, all is fine. When i run in terminal i get this
> error instead:
> 
> File "K:\dev\python\webscraping\kanji_anki.py", line 69, in savefile
>     f.write(card+"\n")
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u898b' in
> position 3 2: ordinal not in range(128)

Are you sure you are writing the same data? That would mean that pydev 
changes the default encoding -- which is evil.

A portable approach would be to use codecs.open() or io.open() instead of 
the built-in:

import io
with io.open(filepath, "a") as f:
    ...

io.open() uses UTF-8 by default, but you can specify other encodings with
io.open(filepath, mode, encoding=whatever).