sqlite utf8 encoding error

[Manlio Perillo]

On 17 Nov 2005 03:47:00 -0800, "Greg Miller" <et1ssgmiller at gmail.com>
wrote:

>I have an application that uses sqlite3 to store job/error data.  When
>I log in as a German user the error codes generated are translated into
>German.  The error code text is then stored in the db.  When I use the
>fetchall() to retrieve the data to generate a report I get the
>following error:
>
>Traceback (most recent call last):
>  File "c:\Pest3\Glosser\baseApp\reportGen.py", line 199, in
>OnGenerateButtonNow
>    self.OnGenerateButton(event)
>  File "c:\Pest3\Glosser\baseApp\reportGen.py", line 243, in
>OnGenerateButton
>    warningresult = messagecursor1.fetchall()
>UnicodeDecodeError: 'utf8' codec can't decode bytes in position 13-18:
>unsupported Unicode code range
>
>does anyone have any idea on what could be going wrong?  The string
>that I store in the database table is:
>
>'Keinen Text für Übereinstimmungsfehler gefunden'
>
>I thought that all strings were stored in unicode in sqlite.
>


No, they are stored as UTF-8 in sqlite and pysqlite has no way to make
sure the string you insert into the database is really encoded in
UTF-8 (the only secure way is to use Unicode strings).

How did you insert that string?

As a partial solution, try to disable automatic conversion of text
fields in Unicode strings:


def convert_text(s):
    # XXX do not use Unicode
    return s


# Register the converter with SQLite
sqlite.register_converter("TEXT", convert_text)


...connect("...",
detect_types=sqlite.PARSE_DECLTYPES|sqlite.PARSE_COLNAMES
)




Regards  Manlio Perillo