Why exception from os.path.exists()?

[eryk sun]

On Sat, Jun 2, 2018 at 11:28 AM, Chris Angelico <rosuav at gmail.com> wrote:
>
> I also can't find anything about path names there. What does POSIX say
> about the concept of relative paths? Does Windows comply with that?

Certainly Windows file-system paths are not POSIX compatible. Seven
path types are supported:

* Extended Local Device (\\?\)
* Local Device (\\.\)
* UNC
* Drive Absolute
* Drive Relative
* Rooted
* Relative

Extended local-device paths only allow backslash as a path separator.
The others allow either backslash or slash. I doubt POSIX would allow
magically reserved DOS device names in every directory or stripping of
trailing dots and spaces from filenames.

But this isn't relevant to NT's POSIX compatibility. A POSIX process
links with psxdll.dll, which connects to the POSIX environment
subsystem (psxss.exe). It gets run from Windows via posix.exe
(console) or psxrun.exe. In the 00s, Microsoft acquired Interix, which
extended the original POSIX subsystem, and integrated it as the
Subsystem for UNIX Applications (SUA). Notably SUA adds a kernel
driver, psxdrv.sys, which facilitates implementing system calls and
signals. There used to be a community website with overviews [1], a
FAQ [2], a forum [3], tool downloads [4], and various documentation
[5]. However, NT's environment subsystems never really had mass
appeal, probably because existing programs had to be ported and
recompiled. SUA is no longer supported as of Windows 8.1 and Server
2012 R2. The community website was closed, and the domain is now held
by a squatter.

Regarding file-system paths, SUA has a single root directory and uses
"/dev/fs/C" for drive "C:" and "/net/server/share" for
"\\server\share".

[1]: http://www.suacommunity.com/SUA_Tools_Env_Start.htm
     https://archive.li/45JG
[2]: http://www.suacommunity.com/FAQs.htm
     https://archive.li/5LFw
[3]: http://www.suacommunity.com/forum2
     https://archive.li/LzZxS
[4]: http://www.suacommunity.com/tool_warehouse.aspx
     https://archive.li/0luI9
[5]: http://www.suacommunity.com/dictionary/fork-entry.php
     https://archive.li/5k8vW

Windows 10 has a Linux subsystem (WSL), but this is not an NT
environment subsystem. WSL processes do not load ntdll.dll. They're
lightweight pico processes with an associated pico provider in the
kernel (lxss.sys, lxcore.sys), and they directly execute native Linux
binaries (no porting and recompiling from source). WSL only supports
the console, but at least the console was upgraded to support
virtual-terminal mode.

> We know that Ctrl-C maps to the internal Windows interrupt
> handler, and "kill process" maps to the internal Windows "terminate",
> but can you send a different process all the different signals and
> handle them differently?

IIRC, the original POSIX subsystem supported only single-threaded
processes, and SIGKILL called NtTerminateThread. Of course the
subsystem has its own client bookkeeping to handle here as well. (For
the Windows subsystem, csrss.exe also maintains shadow process and
thread structures for clients. This is how an environment subsystem
supplements base NT behavior.)

Regarding Ctrl+C, a console session is started by posix.exe, which is
a Windows console application. It translates console control events to
signals, e.g. CTRL_C_EVENT to SIGINT, CTRL_BREAK_EVENT to SIGQUIT, and
otherwise SIGKILL (e.g. closing the console, logoff, shutdown). It
sends the signal number and session ID to the subsystem, which signals
the processes in the given session.

One way for the subsystem to implement signal delivery is via NT's
runtime library function RtlRemoteCall (i.e. suspend the target
thread, get its CPU context and copy it to the stack, modify the
context and stack, and resume). Make a remote call to a known client
function (i.e. in psxdll.dll), which delivers the signal and then
continues the thread's original context via NtContinue. This approach
isn't really efficient, but it's basically how the original POSIX
subsystem worked. SUA probably uses NT asynchronous procedure calls
(APCs).

---

Appendix: NT APCs

NT doesn't have anything exactly like POSIX signals. It has
exceptions, which are handled using either Vectored Exception Handling
or Structured Exception Handling (i.e. MSVC __try, __except,
__finally), and asynchronous procedure calls (APCs). Some POSIX
signals correspond to NT exceptions (e.g. SIGSEGV corresponds to a
STATUS_ACCESS_VIOLATION). But APCs are what a POSIX subsystem would
likely use to implement signals.

There are two types of APC: kernel and user. A thread has an APC queue
for each type. User APCs can be queued from user mode via
NtQueueApcThread, or via WinAPI QueueUserAPC. Some APIs such as
ReadFileEx take an optional completion or notification APC routine,
for which a kernel component queues the user APC.

All APCs have a "kernel routine" and most also have a "normal"
routine. The kernel routine is called first, with the CPU in kernel
mode and its interrupt request level (IRQL) at APC_LEVEL (1). The
kernel routine is passed a pointer to the normal routine, which allows
it to set a different function or none at all (i.e. a NULL pointer).
If it's not NULL, the normal routine is called with the CPU IRQL at
PASSIVE_LEVEL (0), either in kernel mode or user mode, depending on
the APC type.

Kernel APCs are "special" if they're inserted in the queue without a
normal routine. Special APCs get placed ahead of normal APCs in the
queue and can preempt the execution of normal APCs. They're used for
high priority operations. For example, completion of an I/O request
queues a special kernel APC to the thread that originated the request.

Queueing a kernel APC either raises an APC interrupt if the thread is
currently running or awakens the thread if it's currently waiting and
has APC delivery enabled. Queueing a user APC does not raise an
interrupt but may awaken the thread if it's currently in a user-mode
wait and either the wait is alertable or the user APC pending flag is
set.

Kernel APC delivery is triggered either by the APC interrupt handler
or immediately after a context switch to a thread (e.g. when
awakened). User APCs are delivered when returning to user mode, but
only if the thread's user APC pending flag is set. Normally this flag
gets set when the thread does an alertable user-mode delay/wait and
its APC queue isn't empty. It's also set by the NtTestAlert system
call, and also specially set for the user APC that initiates
cross-thread termination.

A thread automatically resumes a delay or wait if it's awakened to
deliver a kernel APC. On the other hand, if a thread is awakened by
queueing a user APC, the wait returns with the status code
STATUS_USER_APC. This is the normal way user APCs are delivered, i.e.
upon returning from an alertable delay or wait system call (e.g.
NtDelayExecution, NtWaitForSingleObject). As mentioned above,
NtTestAlert can also be used to pump the user APC queue.

The APC delivery function first drains the kernel APC queue
completely. If delivering user APCs is enabled (i.e. the user APC
pending flag is set and the previous CPU mode is user mode), it also
delivers the first user APC from the head of the queue. Only one user
APC is delivered because it requires switching to the user-mode APC
dispatcher in ntdll.dll. This can't process the whole queue because,
as discussed above, all APCs have a kernel-mode routine that gets
called first. Thus, with user APCs, the pattern is to call the kernel
routine from the APC delivery function; transition to user-mode to
call the normal routine if the kernel routine didn't set it to NULL;
and then return back to kernel mode via the NtContinue system call.
The latter sets the user APC pending flag to deliver the next user
APC, if any. This cycle continues until the user APC queue is empty.