PEP 383: Non-decodable Bytes in System Character Interfaces

[v+python]

On Apr 21, 11:50 pm, "Martin v. Löwis" <mar... at v.loewis.de> wrote:
> I'm proposing the following PEP for inclusion into Python 3.1.
> Please comment.


Basically the scheme doesn't work.  Aside from that, it is very close.

There are tons of encoding schemes that could work... they don't have
to include half-surrogates or bytes.  What they have to do, is make
sure that they are uniformly applied to all appropriate strings.

The problem with this, and other preceding schemes that have been
discussed here, is that there is no means of ascertaining whether a
particular file name str was obtained from a str API, or was funny-
decoded from a bytes API... and thus, there is no means of reliably
ascertaining whether a particular filename str should be passed to a
str API, or funny-encoded back to bytes.

The assumption in the 2nd Discussion paragraph may hold for a large
percentage of cases, maybe even including some number of 9s, but it is
not guaranteed, and cannot be enforced, therefore there are cases that
could fail.  Whether those failure cases are a concern or not is an
open question.  Picking a character (I don't find U+F01xx in the
Unicode standard, so I don't know what it is) that is obscure, and
unlikely to be used in "real" file names, might help the heuristic
nature of the encoding and decoding avoid most conflicts, but provides
no guarantee that data puns will not occur in practice.  Today's
obscure character is tomorrows commonly used character, perhaps.
Someone not on this list may be happily using that character for their
own nefarious, incompatible purpose.

As I realized in the email-sig, in talking about decoding corrupted
headers, there is only one way to guarantee this... to encode _all_
character sequences, from _all_ interfaces.  Basically it requires
reserving an escape character (I'll use ? in these examples -- yes, an
ASCII question mark -- happens to be illegal in Windows filenames so
all the better on that platform, but the specific character doesn't
matter... avoiding / \ and . is probably good, though).

So the rules would be, when obtaining a file name from the bytes OS
interface, that doesn't properly decode according to UTF-8, decode it
by placing a ? at the beginning, then for each decodable UTF-8
sequence, add a Unicode character -- unless the character is ?, in
which case you add two ??, and for each non-decodable byte sequence,
place a ? and two hex digits, or a ? and a half surrogate code, or a ?
and whatever gibberish you like.  Two hex digits are fine by me, and
will serve for this discussion.

ALSO, when obtaining a file name from the str OS interfaces, encode it
too... if it contains  a ? at the front, it must be replaced by ???
and then any other ? in the name doubled.

Then you have a string that can/must be encoded to be used on either
str or bytes OS interfaces... or any other interfaces that want str or
bytes... but whichever they want, you can do a decode, or determine
that you can't, into that form.  The encode and decode functions
should be available for coders to use, that code to external
interfaces, either OS or 3rd party packages, that do not use this
encoding scheme.  This encoding scheme would be used throughout all
Python APIs (most of which would need very little change to
accommodate it).  However, programs would have to keep track of
whether they were dealing with encoded or unencoded strings, if they
use both types in their program (an example, is hard-coded file names
or file name parts).

The initial ? is not strictly necessary for this scheme to work, but I
think it would be a good flag to the user that this name has been
altered.

This scheme does not depend on assumptions about the use of file
names.

This scheme would be enhanced if the file name APIs returned a subtype
of str for the encoded names, but that should be considered only a
hint, not a requirement.

When encoding file name strings to pass to bytes APIs, the ? followed
by two hex digits would be converted to a byte.  Leading ? would be
dropped, and ?? would convert to ?.  I don't believe failures are
possible when encoding to bytes.

When encoding file name strings to pass to str APIs, the discovery
of ? followed by two hex digits would raise an exception, the file
name is not acceptable to a str API.  However, leading ? would be
dropped, and ?? would convert to ?, and if no ? followed by two hex
digits were found, the file name would be successfully converted for
use on the str API.

Note that not even on Unix/Posix is it particularly easy nor useful to
place a ? into file names from command lines due to shell escapes,
etc.  The use of ? in file names also interferes with easy ability to
specifically match them in globs, etc.

Anything short of such an encoding of both types of interfaces, such
that it is known that all python-manipulated filenames will be
encoded, will have data puns that provide a potential for failure in
edge cases.

Note that in this scheme, no file names that are fully Unicode and do
not contain ? characters are altered by the decoding or the encoding
process.  That will probably reach quite a few 9s of likelihood that
the scheme will go unnoticed by most people and programs and
filenames.  But the scheme will work reliably if implemented correctly
and completely, and will have no edge cases of failure due to not
having data puns.