Unicode support

["Martin v. Löwis"]

Richy2004 wrote:
> NotImplementedError: .readline() is not implemented for UTF-16

As it says: this is, unfortunately, not implemented. Use readlines
instead.

> print (file.read())
[...]
> UnicodeEncodeError: 'charmap' codec can't encode characters in position
> 0-2: character maps to <undefined>

The the .read works perfectly. Don't try to print it, though!
You can only print when the terminal actually supports the characters,
which your terminal doesn't. Try

print repr(file.read())

instead.

> print tokens[0]
[...]
> UnicodeEncodeError: 'charmap' codec can't encode characters in position
> 0-2: character maps to <undefined>

Same issue: As Vincent explains, you can't print ARABIC LETTER NOON
to your terminal, as your terminal simply cannot display that character.

Regards,
Martin