[issue5915] PEP 383 implementation

["Martin v. Löwis" <report@bugs.python.org> at psf.upfronthosting.co.za]

Martin v. Löwis <martin at v.loewis.de> added the comment:

Some notes, in addition to the PEP:

- the file system APIs have all stopped using et converters, and use O&
converters, with a new API function PyUnicode_FSConverter. This outputs
a bytes or bytearray object which needs to be released when done; if it
is a bytearray, the bytes also need to be locked when the GIL is released.

- to convert the environment successfully, initialization of the
filesystemddefaultencoding had to be moved up in the code.

- conversion of the command line arguments can't use codecs. Instead, it
uses the C library, hoping that their encoding is the same as the one
that we later determine as the file system encoding. With C99 mbrtowc,
it is possible to put utf8b escapes in the output; without that
function, every non-ASCII byte gets escaped if the CRT fails to convert
the entire string successfully.

- in the unlikely case that listdir still fails (if the file system
encoding cannot convert ASCII bytes sometimes), listdir now raises an
exception, to be consistent with the failure mode in all other places
where this problem may occur.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5915>
_______________________________________