[Baypiggies] Fwd: [sugar] Database Activity

[Edward Cherlin]

Python UTF-8 issue. Can anybody help?


---------- Forwarded message ----------
From: 7150 <linux.1 at litenverden.com>
Date: Sat, Mar 8, 2008 at 4:02 PM
Subject: Re: [sugar] Database Activity
To: sugar at lists.laptop.org

 This will be my last post on this topic until I learn a bit more.

 It appears that pysqlite has some data integrity issues.

 Interesting link to pysqlite "bug" discussion:
 http://osdir.com/ml/python.db.pysqlite.user/2006-04/msg00020.html

 ---

 "I have built sqlite databases containing utf-8 encoded text.  When I
 access them using pysqlite, utf-8 codepoints appear to be converted to
 latin-1."

 It does this.

 How to stop it?

 An answer on the list was
 (http://osdir.com/ml/python.db.pysqlite.user/2006-04/msg00022.html):

  > This should be:
  >
  >      req.write(elem.encode('utf-8'))
  >
  > write() expects a str object, so what happened
  > in your code was an implicit conversion of your
  > unicode object to a str, doing the encoding using
  > the default system encoding, here 'ascii'.

 The req.write stuff is Greek to me, but I'll see what I can find out
 about it.

 Then from the same post:

 "So, upon reflection, my situation boils down to this:

 (1) A utf-8 encoded string in an sqlite database is queried using
 pysqlite which returns a latin-1 string (a
 gratuitous/silent/unspecified/unrequested conversion).

 . . .

 pysqlite QUESTIONS:

 Question 1:  Why does pysqlite convert the utf-8 sting to latin-1 in the
 *query process* when my sqlite settings are for utf-8?
 i.e., sqlite.h macro: #define SQLITE_UTF8 1

 Question 2:  Does the pysqlite user (programmer) have any control over
 this query/conversion?  (I see nothing in the Python Database API, or
 the pysqlite/sqlite documentation, about either automatic or specified
 conversions.  Also, this conversion does not occur using the sqlite3
 client.)

 Question 3:  Is this a pysqlite bug or a feature, and why?"

 The group had no further answer for the guy.

 ---

 Someone else has noted thet the SQLite3 client does not transform the
 data.  If I learn how to query my UTF-8 database on the XO, I'll let you
 know. But, it's not an XO problem.



 ---

 http://www.litenverden.org
 _______________________________________________
 Sugar mailing list
 Sugar at lists.laptop.org
 http://lists.laptop.org/listinfo/sugar



-- 
Edward Cherlin
End Poverty at a Profit by teaching children business
http://www.EarthTreasury.org/
"The best way to predict the future is to invent it."--Alan Kay