Program chaining on Windows

[Eryk Sun]

On 8/24/20, Rob Cliffe <rob.cliffe at btinternet.com> wrote:
>
> Are you suggesting something I could do in Python that would achieve my
> aim of *replacing* one program by another

No, it's not possible with the Windows API. Implementing POSIX exec
would require extensive use of undocumented NT runtime library
functions, system calls, and PEB data in order to reset the address
space and handle table and reinitialize the process, and it would
probably also require the help of a kernel driver to modify the
_EPROCESS record in kernel space.

>      (1) "*spawn*": I see that there are some os.spawn* functions.

The POSIX way to create a new process is fork (clone), which is
optionally followed by exec (image overlay). Cloning the current
process is fast and efficient. The Windows way to create a new process
is spawn, as implemented by WinAPI CreateProcessW, which is relatively
slow and laborious. In Python, the preferred way to call the latter is
with subprocess.Popen, or helper functions such as subprocess.run.

Spawing a Windows process is a lot of work. It involves creating a new
process object (a large _EPROCESS record in the kernel) and address
space (a VAD tree and page tables); allocating and initializing a
process environment block (PEB; a large block of memory that includes
process flags, parameters, environment variables, loader data,
activation contexts, and much more); optionally inheriting handles for
kernel objects from the parent process; mapping an executable file as
the process image; and creating an initial thread that starts at the
OS entrypoint function (a queued APC that calls LdrInitializeThunk,
which resumes at RtlUserThreadStart). The latter initializes the
process; registers with the session server csrss.exe via its LPC port;
loads and initializes dependent DLLs, which may require multiple
activation contexts and LPC message transactions with the SxS fusion
loader in the session server;  possibly (if it's a console app)
connects to or spawns the console-session host conhost.exe; and calls
the image entrypoint such as __wmainCRTStartup. The latter initializes
the C runtime before calling the application's entrypoint (e.g.
wmain). For Python, the latter calls Py_Main, which initializes the
interpreter before compiling and executing the main script.

>      (2) "*wait*": Wait for what?  How?

Functions such as os.system and subprocess.run wait on (i.e.
synchronize on) the process via WinAPI WaitForSingleObject[Ex], which
waits for the process to terminate.

If the child process uses console I/O and inherits the parent's
console, then it's important for the parent to wait before resuming
its own interactive console UI. Otherwise both parent and child will
compete for the console.

>      (3) "*proxy the exit status*": Sorry, I have no idea what this means.

Generally the parent process needs to know whether the child process
succeeded or failed.  Typically the exit status (aka exit code) for
success is 0, and all other values indicate failure. Commonly an exit
status of 1 is used for failure.

In Windows, the exit status for a process is returned by GetExitCodeProcess.

By "proxy the exit status", I mean that a program that spawns and
waits for another program will usually return the exit status from the
spawned child as its own exit status.

Consider the following snippet from the source code for the py.exe
launcher, which spawns python.exe for the selected version of Python:

    ok = CreateProcessW(NULL, cmdline, NULL, NULL, TRUE,
                        0, NULL, NULL, &si, &pi);
    if (!ok)
        error(RC_CREATE_PROCESS, L"Unable to create process using
'%ls'", cmdline);
    AssignProcessToJobObject(job, pi.hProcess);
    CloseHandle(pi.hThread);
    WaitForSingleObjectEx(pi.hProcess, INFINITE, FALSE);
    ok = GetExitCodeProcess(pi.hProcess, &rc);
    if (!ok)
        error(RC_CREATE_PROCESS, L"Failed to get exit code of process");
    debug(L"child process exit code: %d\n", rc);
    exit(rc);

This C code includes the steps that I discussed: CreateProcessW,
WaitForSingleObjectEx, GetExitCodeProcess, and finally exit(rc), where
rc is the exit status from the spawned python.exe process.

The launcher also creates a kill-on-close, silent-breakaway job object
and assigns the child process to it. That way if the launcher is
terminated or crashes, the python.exe process will also be terminated,
so there's a tight coupling between py.exe and python.exe. The
silent-breakaway setting means processes spawned by python.exe are not
assigned to the job.

You can work with job objects via ctypes or PyWin32's win32job module.
I can provide example code for either approach.

> from the DOS prompt, it works as expected.

You're not running DOS. Most likely the shell you're running is CMD or
PowerShell, attached to a console session that's hosted by either
conhost.exe (default) or openconsole.exe (an open-source build of
conhost.exe that's used by the new Windows Terminal). These are
Windows applications.