Logging library unicode problem

[Vinay Sajip]

On 13 Aug, 11:08, Victor Lin <Borns... at gmail.com> wrote:
> Hi,
> I'm writting a application using python standardloggingsystem. I
> encounter some problem with unicode message passed tologginglibrary.
> I found that unicode message will be messed up bylogginghandler.
>
> piese of StreamHandler:
>
>                 try:
>                     self.stream.write(fs % msg)
>                 except UnicodeError:
>                     self.stream.write(fs % msg.encode("UTF-8"))
>
> It just write the message to stream. If there is some unicode error,
> it would rewrite msg with utf8 encoding.
>
> I write some code to try:
>
>     import sys
>     print u'中文字測試'
>     print sys.stdout.encoding
>     sys.stdout.write(u'中文')
>
> result of that program:
>
> 中文字測試
> cp950
> Traceback (most recent call last):
>   File "update_stockprice.py", line 92, in <module>
>     sys.stdout.write(u'銝剜?')
> UnicodeEncodeError: 'ascii' codec can't encode characters in position
> 0-1: ordin
> al not in range(128)
>
> It show that....
>
> 1. print statement encode what it get with stream.encoding?
> 2. stream.write don't do anything like encoding, just write it
> (because it might be binary data?)
>
> So the problem is : the StreamHandler of standardlogginglibrary use
> stream.write to log message, if there is unicode error, unicode string
> will be encode to utf8. This behavior mess my unicode up.
>
> Here I modify the code of StreamHandler:
>
>                 try:
>                     print >> self.stream, msg
>                     #self.stream.write(fs % msg)
>                 except UnicodeError:
>                     self.stream.write(fs % msg.encode("UTF-8"))
>
> I replace stream.write with print statement, so that it will try to
> use stream.encoding to encode msg. Now everything works fine.
>
> My question is :
> Could the behavior of StreamHandler be considered as a bug?
> If it is, how to report this bug?
> Is my solution correct?
> Are there any side effect will caused by doing so?
> If the code I write is fine, and solve that problem, how to report it
> to Python's project?
> I think this could be helpful for people who also encountered this
> problem.
>
> Thanks.
> Victor Lin.

Hi Victor,

Can you try modifying your patch to use the following logic instead of
the print statement?

if hasattr(self.stream, 'encoding'):
    self.stream.write(fs % msg.encode(self.stream.encoding))
else:
    self.stream.write(fs % msg)

Does this work in your scenario?

Regards,


Vinay Sajip