Bug in python (Weird UnicodeDecodeError)

[Fredrik Lundh]

dbri.tcc at gmail.com wrote:

> I am getting somewhat random UnicodeDecodeError messages in my program.
>
> It is random in that I will be going through a pysqlite database of
> records, manipulate
> the results, and it will throw UnicodeDecodeError apparently without
> regard
> as to what data is being used.
>
> For example I am reading in some book barcodes. These are 7 digit
> strings. It processes a few thousand of these with no problem. But then
> parway through the database results I get something like this:
>
>     for item in list:
> UnicodeDecodeError : 'utf8' codec can't decode bytes in position 26-28:
> invalid data
>
> I am not quite sure how u'2349350' is 26 bytes long. Maybe it is. But
> another time it said something like 'position 129'.
>
> Furthermore when I rearrange the order the database data is retrieved
> (with Order By), it barfs on a different piece of data than it did
> before. I cannot figure it out.

it's probably a bug in pysqlite (or some other C extension), which does
some conversion somewhere, but forgets to check the return status.

(if you raise an exception at the C level, but forget to flag it back to
the interpreter when you return to Python, the error may occur in a
seemingly random location.)

you can usually

    hasattr(None, "none")

to reset the error state (at least this worked in older versions; I think
it should work in 2.3 and 2.4 as well).  try adding such calls after the
database calls, and see if the problem goes away...  (if it does, com-
plain to the pysqlite developers).

</F>