[issue11314] Subprocess suffers 40% process creation overhead penalty

[STINNER Victor]

STINNER Victor <victor.stinner at haypocalc.com> added the comment:

Quick benchmark:
 - fork(), execv(), waitpid(): 19081.1 ms
 - os.popen(): 25769.8 ms
 - subprocess.Popen(): 40025.8 ms
 - subprocess.Popen() patched: 26155.2 ms

I tested Python 2.7 (debug build) on Linux (Debian Sid) with a Pentium 4 (Hyperthreading, 2 fake cores), 2 GB of memory.

Without the patch, subprocess is 55% slower than popen. With the patch, subprocess is 1% slower than popen. It is not a good news: subprocess should be faster before my benchmark tries Popen() without stdout=PIPE ;-)

pitrou> I think your analysis is wrong. These mmap() calls are for
pitrou> anonymous memory, most likely they are emitted by the libc's
pitrou> malloc() to get some memory from the kernel. In other words
pitrou> they will be blazingly fast.

Are you sure? :-)

Note that popen() is 35% slower than fork()+execv()+waitpid(). Patched subprocess is 37% slower than  fork()+execv()+waitpid().

Try attached bench_subprocess.py script to get your own results.

----------
Added file: http://bugs.python.org/file20976/bench_subprocess.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11314>
_______________________________________