[Python-Dev] python process creation overhead

[Gregory Szorc]

I was investigating speeding up Mercurial's test suite (it runs ~13,000
Python processes) and I believe I've identified CPython
process/interpreter creation and destruction as sources of significant
overhead and thus a concern for any CPython user.

Full details are at [1]. tl;dr 10-18% of CPU time in test suite
execution was spent doing the equivalent of `python -c 1` and 30-38% of
CPU time was spent doing the equivalent of `hg version` (I suspect this
is mostly dominated by module importing - and Mercurial has lazy module
importing). My measurements show CPython is significantly slower than
Perl and Ruby when it comes to process/interpreter creation/destruction.
Furthermore, Python 3 appears to be >50% slower than Python 2.

Any system spawning many Python processes will likely feel this
overhead. I'm also a contributor to Firefox's build and testing
infrastructure. Those systems collectively spawn thousands of Python
processes. While I haven't yet measured, I fear that CPython process
overhead is also contributing to significant overhead there.

While more science and knowledge should be added before definite
conclusions are reached, I'd like to ring the figurative bell about this
problem. This apparent inefficiency has me concerned about the use of
CPython in scenarios where process spawning is a common operation and
decent latency is important. This includes CLI tools, build systems, and
testing. I'm curious if others feel this is a problem and what steps can
be taken to mitigate or correct it.

[1] http://www.selenic.com/pipermail/mercurial-devel/2014-May/058854.html

Gregory Szorc
gregory.szorc at gmail.com

P.S. I'm not subscribed, so please CC me on responses.