Unicode from Web to MySQL

[Skip Montanaro]

    Bill> Note that I am able to do create Unicode data and insert it with a
    Bill> carefully controlled unicode string

    Bill> data = u"Make \u0633\u0644\u0627\u0645, not war"
    Bill> c.execute ( INSERT INTO junk (junklet) VALUES ('%s') ''' % 
    Bill> data.encode('utf-8','ignore')

    Bill> but this won't work with what I find on the Web.

I suspect you either don't know the encoding of the data you find on the
web.  Once you know that, you can convert it to unicode, then encode that as
utf-8, placing the result into the database.  You should know the encoding
of the data from the Content-Type header.  If that's missing or incorrect,
you should be able to make a reasonable guess based upon the non-ASCII
patterns you find in the data.

Skip