[Python-Dev] python process creation overhead

[Antoine Pitrou]

Hello,

On Sat, 10 May 2014 13:05:54 -0700
Gregory Szorc <gregory.szorc at gmail.com> wrote:
> I was investigating speeding up Mercurial's test suite (it runs ~13,000
> Python processes) and I believe I've identified CPython
> process/interpreter creation and destruction as sources of significant
> overhead and thus a concern for any CPython user.

This was recently discussed in
https://mail.python.org/pipermail/python-dev/2014-April/134028.html

To give numbers:

$ time python -c pass

real	0m0.012s
user	0m0.008s
sys	0m0.004s

$ time python -c "from mercurial import demandimport; demandimport.enable(); from mercurial.dispatch import request, dispatch; dispatch(request(['version']))"
Mercurial Distributed SCM (version 3.0-rc+2-d924e387604f)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2014 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

real	0m0.056s
user	0m0.044s
sys	0m0.012s

So, even for "hg version", most of the time is spent initializing
Mercurial and its modules, not initializing Python itself. In this
precise measurement, an infinitely fast Python startup would make "hg
version" 20% faster (and other hg commands probably even less, since
they have to load more Mercurial code and data).

I'm wondering why Mercurial couldn't have a transparent daemon mode,
where each "hg" invocation on the CLI would only load whatever
required to communicate with the persistent daemon process. It already
has a commandserver.

> P.S. I'm not subscribed, so please CC me on responses.

I would suggest following the list using the gmane NNTP gateway.

Regards

Antoine.