Changing the (codec) error handler for the stdout/stderr streams in Python 3.0

[Jukka Aho]

Just a tip for those who are only just cutting their teeth on Python 3.0 
and might have encountered the same problem as I did:

When a Python (3.x) program is run on a terminal that only supports a 
legacy character encoding - such as Latin 1 or Codepage 437 - all 
characters printed to stdout will be automatically converted from the 
interpreter's internal Unicode representation to this legacy character 
set.

This is a nice feature to have, of course, but if the original Unicode 
string contains characters for which there is no equivalent in the 
terminal's legacy character set, you will get the dreaded 
"UnicodeEncodeError" exception.

In other words, the "sys.stdout" stream - as well as the "sys.stderr" 
stream - have both been hardwired to do their character encoding magic, 
by default, using the 'strict' error handling scheme:

--- 8< ---

 >>> import sys
 >>> sys.stdout.errors
 'strict'
 >>> sys.stderr.errors
 'strict'

--- 8< ---

So, essentially, printing out anything but ASCII to stdout is not really 
safe in Python... unless you know beforehand, for sure, what characters 
the terminal will support - which at least in my mind kind of defeats 
the whole purpose of those automatic, implicit conversions.

Now, I have written a more flexible custom error handler myself and 
registered it with Python's codec system, using the 
codecs.register_error() function. When the handler encounters a 
problematic codepoint, it will either suggest a "similar-enough" Latin 1 
or ASCII substitution for it, or if there is none available in its 
internal conversion table, it will simply print it out using the U+xxxx 
notation. The "UnicodeEncodeError" exception will never occur with it.

Instead of creating a custom error handler from scratch one could also 
make use of one of Python's built-in, less restrictive error handlers, 
such as 'ignore', 'replace', 'xmlcharrefreplace', or 'backslashreplace'.

But in order to make things work as transparently and smoothly as 
possible, I needed a way to make both the "sys.stdio" and "sys.stderr" 
streams actually _use_ my custom error handler, instead of the default 
one.

Unfortunately, the current implementation of io.TextIOWrapper (in Python 
3.0b2, at least) does not yet offer a public, documented interface for 
changing the codec error handler - or, indeed, the target encoding 
itself - for streams that have already been opened, and this means you 
can't "officially" change it for the "stdout" or "stderr" streams, 
either. (The need for this functionality is acknowledged in PEP-3116, 
but has apparently not been implemented yet. [1])

So, after examining io.py and scratching my head a bit, here's how one 
can currently hack one's way around this limitation:

--- 8< ---

 import sys
 sys.stdout._errors = 'backslashreplace'
 sys.stdout._get_encoder()
 sys.stderr._errors = 'backslashreplace'
 sys.stderr._get_encoder()

--- 8< ---

Issuing these commands makes printing out Unicode strings to a legacy 
terminal a safe procedure again and you're not going get unexpected 
"UnicodeEncodeErrors" thrown in your face any longer. (Note: 
'backslashreplace' is just an example here; you could substitute the 
error handler of your choice for it.)

The downside of this solution is, of course, that it will break down if 
the private implementation of io.TextIOWrapper in io.py changes in the 
future. But as a workaround, I feel it is sufficient for now, while 
waiting for the "real" support to appear in the library.

(If there's a cleaner and more future-proof way of doing the same thing 
right now, I'd of course love to hear about it...)

_____

1. http://mail.python.org/pipermail/python-3000/2008-April/013366.html

-- 
znark