Convertion of Unicode to ASCII NIGHTMARE

[Robert Kern]

ChaosKCW wrote:
> Hi
> 
> I am reading from an oracle database using cx_Oracle. I am writing to a
> SQLite database using apsw.
> 
> The oracle database is returning utf-8 characters for euopean item
> names, ie special charcaters from an ASCII perspective.

I'm not sure that you are using those terms correctly. From your description
below, it seems that your data is being returned from the Oracle database as
unicode strings rather than regular strings containing UTF-8 encoded data. These
European characters are not "special characters from an ASCII perspective;" they
simply aren't characters in the ASCII character set at all.

> I get the following error:
> 
>>   SQLiteCur.execute(sql, row)
>>UnicodeDecodeError: 'ascii' codec can't decode byte 0xdc in position 12: ordinal not in >range(128)
> 
> I have googled for serval days now and still cant get it to encode to
> ascii.

Don't. You can't. Those characters don't exist in the ASCII character set.
SQLite 3.0 deals with UTF-8 encoded SQL statements, though.

http://www.sqlite.org/version3.html

> I encode the SQL as follows:
> 
>         sql = "insert into %s values %s" % (SQLiteTable, paramstr)
>         sql.encode('ascii', 'ignore')

The .encode() method returns a new value; it does not change an object inplace.

  sql = sql.encode('utf-8')

-- 
Robert Kern
robert.kern at gmail.com

"I have come to believe that the whole world is an enigma, a harmless enigma
 that is made terrible by our own mad attempt to interpret it as though it had
 an underlying truth."
  -- Umberto Eco