[Python-Dev] PEP 383 update: utf8b is now the error handler

[Stephen J. Turnbull]

Zooko O'Whielacronx writes:

 > How would an application make sure that they were producing only
 > valid unicode?

That's very difficult.  There are a couple of sources that I can think
of, in Python: C modules, chr(), \u literals, and now codecs with the
'utf8b'.  There may be others.  You'd need to review your own code for
all of them very carefully, and you'd have to validate all strings
returned by non-validating APIs (which is all of them in Python now,
although many of them can probably be trusted, such as codecs not
using the 'utf8b' error handler).

 > How about add an option to os.listdir() named "errors" with default
 > value 'utf8b'

Seems reasonable to me, but Martin's probably thought more carefully
about it.  I don't think its applicable to your use case, though,
because you want to be able to *access* those files as well as display
the names to the users, right?  You won't be able to access those
files if you receive the names already munged by the error handler.