[Pythonmac-SIG] Chinese glyphs in Python 3
Ronald Oussoren
ronaldoussoren at mac.com
Wed May 27 14:28:12 CEST 2009
--===============1820434273==
Content-type: multipart/signed; boundary=Apple-Mail-82-903528184; micalg=sha1;
protocol="application/pkcs7-signature"
--Apple-Mail-82-903528184
Content-Type: multipart/alternative;
boundary=Apple-Mail-81-903528086
--Apple-Mail-81-903528086
Content-Type: text/plain;
charset=US-ASCII;
format=flowed;
delsp=yes
Content-Transfer-Encoding: 7bit
I've found the source of this bug: Tcl/Tk's rendering of Unicode data
is broken on OSX. Luckily this is fixed on Tcl/Tk 8.5, but that
doesn't help you very much because the Python installers all link to
the (system install of) Tcl/Tk 8.4 and that version is not binary
compatible with the later version.
Which brings the question: is there anyone on the list that would like
to look into providing two copies of Tkinter in the binary installers
on OSX? This would preferably install both copies of the tkinter
extension and magicly select the right copy to use at runtime. That
way Tkinter, and more importantly IDLE, would work out of the box and
anyone that would need a better version of Tk can use that by
installing Tk themself.
I don't have time to work on this myself though.
Ronald
On 26 May, 2009, at 8:56, Ronald Oussoren wrote:
> I've filed this as issue 6109 at the python bugtracker (http://bugs.python.org/issue6109
> ). I haven't uploaded your example file yet, would you mind if I did
> upload the file to the tracker?
>
> Ronald
>
> On 26 May, 2009, at 2:27, John Newman wrote:
>
>> Sorry, pasting the Chinese text into email wasn't very clever. I've
>> attached a txt file with the Chinese saved as utf-8.
>>
>> John
>>
>>
>> -----Original Message-----
>> From: Ronald Oussoren [mailto:ronaldoussoren at mac.com]
>> Sent: Mon 5/25/2009 11:30 AM
>> To: John Newman
>> Cc: pythonmac-sig at python.org
>> Subject: Re: [Pythonmac-SIG] Chinese glyphs in Python 3
>>
>>
>> On 25 May, 2009, at 16:36, John Newman wrote:
>>
>>
>> Ronald
>>
>> See the attachment, where most of the Chinese glyphs are
>> correctly displayed, but not all.
>>
>> When I copy and paste these glyphs from Python 3 to JEdit,
>> all the glyphs display correctly, so the correct unicode code
>> points are there underlying the glyphs. I've tried most of the
>> fonts available to me in the GUI and I still can't get them all to
>> display properly.
>>
>>
>> Could you post the actual text-file as well?
>>
>> Ronald
>>
>>
>> John
>>
>>
>> -----Original Message-----
>> From: Ronald Oussoren [mailto:ronaldoussoren at mac.com]
>> Sent: Sun 5/24/2009 11:48 PM
>> To: John Newman
>> Cc: pythonmac-sig at python.org
>> Subject: Re: [Pythonmac-SIG] Chinese glyphs in Python 3
>>
>> John,
>>
>> On 25 May, 2009, at 1:34, John Newman wrote:
>>
>>
>> Apologies if my question is ridiculously trivial -
>> I'm not a developer, just a relatively new user of Python 3.0.1 on
>> Mac OS 10.4.11. and I have only just now joined the list.
>>
>> When I read in a Chinese text in the Python 3 IDLE
>> GUI on Windows XP I see all the Chinese glyphs displayed properly.
>> On my Mac, about one third of the Chinese characters are not
>> displayed correctly (just empty or black boxes in place of glyphs).
>> It doesn't matter whether I save/open texts as utf-8, utf-16, etc.
>> [The glyphs display fine in TextEdit, JEdit, Word etc.] And the
>> same kind of problem occurs when I scroll through the list of font
>> names in Preferences in IDLE: the names in Chinese glyphs have a
>> number of white or black boxes instead of the glyphs.
>>
>> I just assume that this has something to do with
>> the locale settings in IDLE? On my machine:
>>
>> >>> locale.getpreferredencoding()
>> 'mac-roman'
>> >>> locale.getlocale()
>> (None, None)
>>
>> In my Windows XP, the locale settings are (English,
>> '1252') and I presume that this difference is relevant to
>> understanding the different effects I get opening Chinese texts in
>> my Windows XP and my Mac. 'mac-roman' would not be my natural
>> choice of encoding if I am looking at Chinese text! I need an
>> encoding which can handle the range of glyphs we find in GB 18030,
>> say.
>>
>> Am I being naive in thinking that all I have to do
>> in Python is somehow change the locale settings in some way which
>> will display Chnese glyphs? I'm at a loss to know what I should do
>> in order to display Chinese glyphs properly on the Mac. I tried
>> experimenting with "setlocale" but couldn't make progress.
>>
>> Any suggestions would be very welcome.
>>
>>
>>
>> This might be a font issue, although the default font
>> (Courier) seems to be capable of displaying unicode text and
>> therefore saving as UTF-8 should work. Another possible souce for
>> this problem is the GUI framework used by IDLE.
>>
>> Could you post an example of a file that shows the problem?
>>
>> Ronald
>>
>>
>>
>>
>>
>> <glyphs.jpg>
>>
>>
>>
>>
>>
>> <Chinese.txt>
>
> _______________________________________________
> Pythonmac-SIG maillist - Pythonmac-SIG at python.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig
--Apple-Mail-81-903528086
Content-Type: text/html;
charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
<html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; "><div><br></div><div>I've found =
the source of this bug: Tcl/Tk's rendering of Unicode data is broken on =
OSX. Luckily this is fixed on Tcl/Tk 8.5, but that doesn't help you very =
much because the Python installers all link to the (system install of) =
Tcl/Tk 8.4 and that version is not binary compatible with the later =
version.</div><div><br></div><div>Which brings the question: is there =
anyone on the list that would like to look into providing two copies of =
Tkinter in the binary installers on OSX? This would preferably install =
both copies of the tkinter extension and magicly select the right copy =
to use at runtime. That way Tkinter, and more importantly IDLE, would =
work out of the box and anyone that would need a better version of Tk =
can use that by installing Tk themself. =
</div><div><br></div><div>I don't have time to work on this myself =
though.</div><div><br></div><div>Ronald</div><div><br></div><div><div>On =
26 May, 2009, at 8:56, Ronald Oussoren wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; ">I've filed this as issue 6109 =
at the python bugtracker (<a =
href=3D"http://bugs.python.org/issue6109">http://bugs.python.org/issue6109=
</a>). I haven't uploaded your example file yet, would you mind if I did =
upload the file to the =
tracker?<div><br></div><div>Ronald<br><div><br><div><div>On 26 May, =
2009, at 2:27, John Newman wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"> <div> =
<!-- Converted from text/plain format --><p><font size=3D"2">Sorry, =
pasting the Chinese text into email wasn't very clever. I've attached a =
txt file with the Chinese saved as utf-8.<br> <br> John<br> <br> <br> =
-----Original Message-----<br> From: Ronald Oussoren [<a =
href=3D"mailto:ronaldoussoren at mac.com">mailto:ronaldoussoren at mac.com</a>]<=
br> Sent: Mon 5/25/2009 11:30 AM<br> To: John Newman<br> Cc: <a =
href=3D"mailto:pythonmac-sig at python.org">pythonmac-sig at python.org</a><br> =
Subject: Re: [Pythonmac-SIG] Chinese glyphs in Python 3<br> <br> <br> On =
25 May, 2009, at 16:36, John Newman wrote:<br> <br> <br> =
Ronald<br> =
<br> =
See the attachment, where =
most of the Chinese glyphs are correctly displayed, but not all.<br> =
<br> =
When I copy and paste these =
glyphs from Python 3 to JEdit, all the glyphs display correctly, so the =
correct unicode code points are there underlying the glyphs. I've tried =
most of the fonts available to me in the GUI and I still can't get them =
all to display properly.<br> =
<br> <br> Could you post the =
actual text-file as well?<br> <br> Ronald<br> <br> =
<br> =
John<br> =
<br> =
<br> =
-----Original =
Message-----<br> From: Ronald =
Oussoren [<a =
href=3D"mailto:ronaldoussoren at mac.com">mailto:ronaldoussoren at mac.com</a>]<=
br> Sent: Sun 5/24/2009 11:48 =
PM<br> To: John Newman<br> =
Cc: <a =
href=3D"mailto:pythonmac-sig at python.org">pythonmac-sig at python.org</a><br> =
Subject: Re: [Pythonmac-SIG] =
Chinese glyphs in Python 3<br> =
<br> =
John,<br> =
<br> =
On 25 May, 2009, at 1:34, =
John Newman wrote:<br> <br> =
<br> =
=
Apologies if my question is =
ridiculously trivial - I'm not a developer, just a relatively new user =
of Python 3.0.1 on Mac OS 10.4.11. and I have only just now joined the =
list.<br> =
<br> =
=
When I read in a Chinese text =
in the Python 3 IDLE GUI on Windows XP I see all the Chinese glyphs =
displayed properly. On my Mac, about one third of the Chinese characters =
are not displayed correctly (just empty or black boxes in place of =
glyphs). It doesn't matter whether I save/open texts as utf-8, utf-16, =
etc. [The glyphs display fine in TextEdit, JEdit, Word etc.] And =
the same kind of problem occurs when I scroll through the list of font =
names in Preferences in IDLE: the names in Chinese glyphs have a number =
of white or black boxes instead of the glyphs.<br> =
=
<br> =
=
I just assume that this has =
something to do with the locale settings in IDLE? On my machine:<br> =
=
<br> =
=
>>> =
locale.getpreferredencoding()<br> =
=
'mac-roman'<br> =
=
>>> =
locale.getlocale()<br> =
(None, None)<br> =
=
<br> =
=
In my Windows XP, the locale =
settings are (English, '1252') and I presume that this difference is =
relevant to understanding the different effects I get opening Chinese =
texts in my Windows XP and my Mac. 'mac-roman' would not be my natural =
choice of encoding if I am looking at Chinese text! I need an encoding =
which can handle the range of glyphs we find in GB 18030, say.<br> =
=
<br> =
=
Am I being naive in thinking =
that all I have to do in Python is somehow change the locale settings in =
some way which will display Chnese glyphs? I'm at a loss to know =
what I should do in order to display Chinese glyphs properly on the Mac. =
I tried experimenting with "setlocale" but couldn't make progress.<br> =
=
<br> =
=
Any suggestions would be very =
welcome.<br> =
<br> =
<br> =
<br> =
This might be a font issue, =
although the default font (Courier) seems to be capable of displaying =
unicode text and therefore saving as UTF-8 should work. Another =
possible souce for this problem is the GUI framework used by IDLE.<br> =
<br> =
Could you post an =
example of a file that shows the problem?<br> =
<br> =
Ronald<br> =
<br> =
<br> =
<br> =
<br> <br> =
<glyphs.jpg><br> <br> =
<br> <br> <br> </font> </p> </div> =
<span><Chinese.txt></span></blockquote></div><br></div></div></div>_=
______________________________________________<br>Pythonmac-SIG maillist =
- <a =
href=3D"mailto:Pythonmac-SIG at python.org">Pythonmac-SIG at python.org</a><br><=
a =
href=3D"http://mail.python.org/mailman/listinfo/pythonmac-sig">http://mail=
.python.org/mailman/listinfo/pythonmac-sig</a><br></blockquote></div><br><=
/body></html>=
--Apple-Mail-81-903528086--
--Apple-Mail-82-903528184
Content-Disposition: attachment;
filename=smime.p7s
Content-Type: application/pkcs7-signature;
name=smime.p7s
Content-Transfer-Encoding: base64
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIFPzCCBTsw
ggMjoAMCAQICAwQdWTANBgkqhkiG9w0BAQUFADB5MRAwDgYDVQQKEwdSb290IENBMR4wHAYDVQQL
ExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBBdXRob3Jp
dHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0wNzEwMDQxNDQ3MjhaFw0w
OTEwMDMxNDQ3MjhaMEExGDAWBgNVBAMTD1JvbmFsZCBPdXNzb3JlbjElMCMGCSqGSIb3DQEJARYW
cm9uYWxkb3Vzc29yZW5AbWFjLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOwe
sF7LDCvCwKd+kX0mG+60zixF28kOCZ6NDlTMUTNmaGTTpPypY4DjZIWRKrcKhnwN9m7Qe4kIRq8W
sNw6KGO1hg2xXspOmA19gadEMILN41Zi4UFPLt3DJv36kTE4ZEZPn3lJtLL0/1kK2Qrnfo16Rnoo
wfh4E4qBO9xdbUVXhbsC4Ou5YSC9oGfRveDR6ondmVcToM1oGS9ycPleJx0JgOPKBpDFG8xMdMD1
bnBolVof+19yIdKcuQKTWgayrrqHzpnhe2Ue2+6XJNgXEs92hSjBKtMr+xCppM38OesTpyzFxvMb
iAVEzyWkRrKL4yuTmoSR4LOPAmm/CCPdQB8CAwEAAaOCAQIwgf8wDAYDVR0TAQH/BAIwADBWBglg
hkgBhvhCAQ0ESRZHVG8gZ2V0IHlvdXIgb3duIGNlcnRpZmljYXRlIGZvciBGUkVFIGhlYWQgb3Zl
ciB0byBodHRwOi8vd3d3LkNBY2VydC5vcmcwQAYDVR0lBDkwNwYIKwYBBQUHAwQGCCsGAQUFBwMC
BgorBgEEAYI3CgMEBgorBgEEAYI3CgMDBglghkgBhvhCBAEwMgYIKwYBBQUHAQEEJjAkMCIGCCsG
AQUFBzABhhZodHRwOi8vb2NzcC5jYWNlcnQub3JnMCEGA1UdEQQaMBiBFnJvbmFsZG91c3NvcmVu
QG1hYy5jb20wDQYJKoZIhvcNAQEFBQADggIBAHvpWk6DZ6OGMDcyOWPlYhuXNVftnlz8jZNUKzDh
b+zLKNd40GC2nN2sZltZ2NJLItN1DblSpWh7Y151e3mYJEYnWWtpiDUB0C6J8lS7R1BS1IY6NJNj
FC/i4FRc8f/TlMizcNUipB2qXDPUQxb4IIlTq8T12lHeSlbUzueiJaEIqnWxV1PZPD7gLJKo+x+g
j5cjk1KHiQgHgKcuS4E6O+eiVlijJ/I3+QF6q+hcHdGS+Uo5VVUjFVxsVQ7Zatumq/Gyj1ly8yr9
KQAO9swqEutHF6CfzKQ5cGNvBiYWuOc2XbRIElRNVgvnYz/ysaVGLqssj3iNN3+h8F0EI+EpUMt3
TMDYo78GZGe6no0VizOz34uHvIu0hH8CDet6ZzCy2iodGCyyzGQx51XO0g2r3BYbhY39JfbyJU7m
W3QK33yKHW8gACAARNuIwJOJiydyGbAkmsg7csHcxAk9ICrgPW8xWWmswOhPYG2sBkVtRFoZ5MT0
h8oIMUY3T0GFpTwvtKBJcNjPjloSR9FlwlgbxvcaypqFnT/FrPX0N9lCZCzSu8tUjiqns1vjOEgF
x5RUY9ouAsmOdXrHNC3lfiwfD2pAKvDYSgEM5cJAqQjiUDdL7Y3P6M3rVjuUYdSPvS5zU3hCQtm+
dTb5q2QNhgHq1PI35MQfaT+mXF8Yn9LsyK2tMYIDMzCCAy8CAQEwgYAweTEQMA4GA1UEChMHUm9v
dCBDQTEeMBwGA1UECxMVaHR0cDovL3d3dy5jYWNlcnQub3JnMSIwIAYDVQQDExlDQSBDZXJ0IFNp
Z25pbmcgQXV0aG9yaXR5MSEwHwYJKoZIhvcNAQkBFhJzdXBwb3J0QGNhY2VydC5vcmcCAwQdWTAJ
BgUrDgMCGgUAoIIBhzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0w
OTA1MjYxNTIwMTdaMCMGCSqGSIb3DQEJBDEWBBQ02hYFbyiYRkO/sASF7AbFuAYV4TCBkQYJKwYB
BAGCNxAEMYGDMIGAMHkxEDAOBgNVBAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2Fj
ZXJ0Lm9yZzEiMCAGA1UEAxMZQ0EgQ2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJ
ARYSc3VwcG9ydEBjYWNlcnQub3JnAgMEHVkwgZMGCyqGSIb3DQEJEAILMYGDoIGAMHkxEDAOBgNV
BAoTB1Jvb3QgQ0ExHjAcBgNVBAsTFWh0dHA6Ly93d3cuY2FjZXJ0Lm9yZzEiMCAGA1UEAxMZQ0Eg
Q2VydCBTaWduaW5nIEF1dGhvcml0eTEhMB8GCSqGSIb3DQEJARYSc3VwcG9ydEBjYWNlcnQub3Jn
AgMEHVkwDQYJKoZIhvcNAQEBBQAEggEACfPwp+iJU2R3YKA2FdnaCImcXwIwfiQ++OMwryxCuJ2/
G6CsyhS/nBVPQBjG7lYyqcD87bmA2cg/PLTQAdRMUTI5drM76LE+Cfw7c8tw/khAtHnAm8yuQu2L
XVcbNslIPPNJzfkogki3KFU3Y0mhLylrZ/K6BSl0MKPAZTXFx26LtPvoWMT817GF/+yosqUGJbJK
ZThCDV4figJNR8JEj+GO+/FQMKlAurk/zJck48eDy3SmvHwBr1xTbIXTNI9NO4AuH3PXlkKS/wq5
pNDp3dicn5ZXkotUdRGwhBS6iwUbRCKMmiCmBSjY85xHaZA3qaU4StGErw6U9+A5y3ME0wAAAAAA
AA==
--Apple-Mail-82-903528184--
--===============1820434273==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
Pythonmac-SIG maillist - Pythonmac-SIG at python.org
http://mail.python.org/mailman/listinfo/pythonmac-sig
--===============1820434273==--
More information about the Pythonmac-SIG
mailing list