Convertion of Unicode to ASCII NIGHTMARE

[Paul Boddie]

Robert Kern wrote:
> Roger Binns wrote:
> > "Paul Boddie" <paul at boddie.org.uk> wrote
> >>It looks like you may have Unicode objects that you're presenting to
> >>sqlite. In any case, with earlier versions of pysqlite that I've used,
> >>you need to connect with a special unicode_results parameter,

Note that I've since mentioned client_encoding which seems to matter
for pysqlite 1.x.

> > He is using apsw.  apsw correctly handles unicode.  In fact it won't
> > accept a str with bytes >127 as they will be an unknown encoding and
> > SQLite only uses Unicode internally.  It does have a blob type
> > using buffer for situations where binary data needs to be stored.
> > pysqlite's mishandling of Unicode is one of the things that drove
> > me to writing apsw in the first place.

For pysqlite 2.x, it appears that Unicode objects can be handed
straight to the API methods, and I'd be interested to hear about your
problems with pysqlite, Unicode and what actually made you write apsw
instead.

> Ah, I misread the OP's traceback.
>
> Okay, the OP is getting regular strings, which are probably encoded in
> ISO-8859-1 if I had to guess, from the Oracle DB. He is trying to pass them in
> to SQLiteCur.execute() which tries to make a unicode string from the input:

[...]

There's an Oracle environment variable that appears to make a
difference: NLS_CHARSET, perhaps - it's been a while since I've had to
deal with Oracle, and I'm not looking for another adventure into
Oracle's hideous documentation to find out.

> *Now*, my advice to the OP is to figure out the encoding of the strings that are
> being returned from Oracle. As I said, ISO-8859-1 is probably a good guess.
> Then, he would *decode* the string to a unicode string using the encoding. E.g.:
>
>   row = row.decode('iso-8859-1')
>
> Then everything should be peachy. I hope.

Yes, just find out what Oracle wants first, then set it all up, noting
that without looking into the Oracle wrapper being used, I can't
suggest an easier way.

Paul