How to suppress UnicodeError exception when enabling httplib debug?

[Baptiste Lepilleur]

I activated httplib debug, and when trace are printed, a UnicodeError 
exception is thrown. I have already set sys.stdout to use utf-8 encoding 
(this removed the exception when *I* was printing unicode), but from the 
stacktrace below, the encoding seems to magically have switched to 'ascii' 
when httplib does the printing...

import codecs
import sys
sys.stdout = codecs.getwriter("utf-8")(sys.__stdout__)
...
def fetchURL( url ):
    request = urllib2.Request( url )
    opener = urllib2.build_opener( urllib2.HTTPHandler(debuglevel=1) )
    feeddata = opener.open(request)
    data = feeddata.read()
    return data.decode( 'utf-8', 'replace' )
...

Here is the traceback exceprt:

  File "updater.py", line 120, in getStoryChapter
    content = fetchURL( url )
  File "updater.py", line 43, in fetchURL
    feeddata = opener.open(request)
  File "D:\python24\lib\urllib2.py", line 358, in open
    response = self._open(req, data)
  File "D:\python24\lib\urllib2.py", line 376, in _open
    '_open', req)
  File "D:\python24\lib\urllib2.py", line 337, in _call_chain
    result = func(*args)
  File "D:\python24\lib\urllib2.py", line 1021, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "D:\python24\lib\urllib2.py", line 994, in do_open
    r = h.getresponse()
  File "D:\python24\lib\httplib.py", line 863, in getresponse
    response.begin()
  File "D:\python24\lib\httplib.py", line 365, in begin
    print "header:", hdr,
  File "D:\python24\lib\codecs.py", line 178, in write
    data, consumed = self.encode(object, self.errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x89 in position 840: 
ordinal not in range(128)

Why is the print statement using the 'ascii' codec instead of utf-8? Is 
there a way to ensure print always work (I'm just using it for debugging 
purpose) ?


I'm using python 2.4.2.

Baptiste.