usage of <string>.encode('utf-8','xmlcharrefreplace')?

[J Peyret]

On Feb 18, 10:54 pm, 7stud <bbxx789_0... at yahoo.com> wrote:

> One last point: you can't display a unicode string.  The very act of
> trying to print a unicode string causes it to be converted to a
> regular string.  If you try to display a unicode string without
> explicitly encode()'ing it first, i.e. converting it to a regular
> string using a specified secret code--a so called 'codec', python will
> implicitly attempt to convert the unicode string to a regular string
> using the default codec, which is usually set to ascii.

Yes, the string above was obtained by printing, which got it into
ASCII format, as you picked up.
Something else to watch out for when posting unicode issues.

The solution I ended up with was

1) Find out the encoding in the data file.

In Ubuntu's gedit editor, menu 'Save As...' displays the encoding at
the bottom of the save prompt dialog.

ISO-8859-15 in my case.

2) Look up encoding corresponding to ISO-8859-15 at

http://docs.python.org/lib/standard-encodings.html

3) Applying the decode/encode recipe suggested previously, for which I
do understand the reason now.

#converting rawdescr
#from ISO-8859-15 (from the file)
#to UTF-8 (what postgresql wants)
#no error handler required.
decodeddescr = rawdescr.decode('iso8859_15').encode('utf-8')

postgresql insert is done using decodeddescr variable.

Postgresql is happy, I'm happy.