[issue35537] use os.posix_spawn in subprocess

[STINNER Victor]

STINNER Victor <vstinner at redhat.com> added the comment:

"posix_spawn() is faster"

This assumption needs a benchmark!

I ran a benchmark on Linux (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28) using attached subprocess_bench.py.

I rebased the PR 11242 on master and run the benchmark in virtual environment:

git co pr/11242
# current branch: PR 11242
make distclean
./configure
make
./python -m venv env
env/bin/python -m pip install perf
env/bin/python subprocess_bench.py -v -o posix_spawn.json
git co master
env/bin/python subprocess_bench.py -v -o fork_exec.json
env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 

The result is quite explicit: the PR makes subprocess.Popen 61x faster!!! 

$ env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 27.1 ms +- 0.4 ms -> [posix_spawn] 447 us +- 163 us: 60.55x faster (-98%)

That's the best case:

* The parent process (Python) allocated 2 GiB of memory: that's not uncommon for large application. On OpenStack for example, it's common that a process takes more than 1 GiB.
* The child process has a short execution time and allocates few memory.


On my config (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28), os.posix_spawn() uses vfork():

$ strace -o trace ./python -c 'import subprocess; subprocess.run(["/bin/true"], close_fds=False, restore_signals=False)'
$ grep clone trace
clone(child_stack=0x7fab28ac0ff0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 23073

I guess that the 61x speedup mostly comes from vfork().

See also bpo-34663 for previous discussion about vfork() and os.posix_spawn(). In short, the glibc is smart and detects when vfork() can be used or not.

----------
Added file: https://bugs.python.org/file48011/subprocess_bench.py

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue35537>
_______________________________________