Why exception from os.path.exists()?

[Steven D'Aprano]

On Tue, 05 Jun 2018 17:28:24 +0200, Peter J. Holzer wrote:
[...]
> If a disk with a file system which allows embedded NUL characters is
> mounted on Linux (let's for the sake of the argument assume it is HFS+,
> although I have to admit that I don't know anything about the internals
> of that filesystem), then the low level filesystem code has to map that
> character to something else. Even the generic filesystem code of the
> kernel will never see that NUL character,

Even if this were true, why is it even the tiniest bit relevant to what 
os.path.exists() does when given a path containing a NUL byte?


> let alone the user space. As
> far as the OS is concerned, that file doesn't contain a NUL character.

I don't care about "as far as the OS". I care about users, people like 
me. If I say "Here's a file called "sp\0am" then I don't care what the OS 
does, or the FS driver, or the disk hardware. I couldn't care less what 
the actual byte pattern on the disk is.

If you told me that the pattern of bytes representing that filename was 
0x0102030405 then I'd be momentarily impressed by the curious pattern and 
then do my best to immediately forget all about it.

As a Python programmer, *why do you care* about NULs? How does this 
special treatment make your life as a Python programmer better?


> The whole system (except for some low-level FS-dependent code) will
> always only see the mapped name.

Yes. So what? That's *already the case*. Even Python string you pass to 
os.path.exists is already mapped, and errors from the kernel are mapped 
to False. Why should NUL be treated differently?

Typical Linux file systems (ext3, ext4, btrfs, ReiserFS etc) don't 
support Unicode, only bytes 0...255, but we can query "invalid" file 
names containing characters like δ ж or ∆, without any problem. We don't 
get ValueError just because of some irrelevant technical detail that the 
file system doesn't support characters outside of the range of bytes 
1...255 (excluding 47). We can do this because Python seamlessly maps 
Unicode to bytes and back again.

You may have heard of a little-known operating system called "Windows", 
which defaults to NTFS as its file system. I'm told that there are a few 
people who use this file system. Even under Linux, you might have 
(knowingly or unknowingly) used a network file system or storage device 
that used NTFS under the hood.

If so, then every time you query a filename, even an ordinary looking one 
like "foo", you could be dealing with multiple NUL bytes, as the NTFS 
file system (even under Linux!) uses Unicode file names encoded with 
UTF-16. There's a good chance that EVERY filename you've used on a NAS 
device or network drive has included embedded NUL bytes.

You've painted a pretty picture of the supposed confusion and difficulty 
such NUL bytes would cause, but its all nonsense. We already can 
seamlessly and transparently interact with file systems where file names 
include NUL bytes under Linux.

BUT even if what you said was true, that Linux cannot deal with NUL bytes 
in file names even with driver support, even if passing a NUL byte to the 
Linux kernel would cause the fall of human civilization, that STILL 
wouldn't require us to raise ValueError from os.path.exists!




-- 
Steven D'Aprano
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson