[Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)

[Daniel Holth]

I find Python's startup time to be very sluggish. I wish it was less
than 50 milliseconds (0.05 seconds) including running hg, which is the
common threshold for "instant". On my machine 'python -c ""' takes
about 0.05 seconds but 'python3 -c ""' takes 0.125 seconds. I will be
very happy to see any speedup.

On Tue, Apr 15, 2014 at 11:34 AM, Skip Montanaro <skip at pobox.com> wrote:
> Eric wrote:
>
>> Perhaps not so much "a very real advantage" as "less of a
>> distraction".  It's still significantly slower than 2.7.  :)
>
> I'm still confused. I peeked in /usr/bin/hg. The only "system" modules
> it imports directly are os and system (maybe I'm using an ancient
> version?). After that, it imports its own lazy import module. This
> suggests to me that Mercurial's import slowness is mostly in its own
> code (I counted 104 Python modules and six shared objects in its
> mercurial package, which isn't going to be affected (much?) by
> freezing the Python standard modules.
>
> I'm not trying to be difficult here. I thought that way back in the
> day a huge amount of work was done to remove needless filesystem
> activity, and zip packaging has been around for quite awhile.
>
> As an experiment, I ran hg pull as
>
>     /usr/bin/python -v /usr/bin/hg pull
>
> in my cpython repo then looked at the -v output. Summarizing the
> output I saw the following:
>
>     30 imports (0 dlopens)
>
>     Python banner printed
>
>     86 imports (18 dlopens)
>
>     adding changesets message
>
>     5 imports (2 dlopens)
>
>     adding manifests message
>
>     1 import (0 dlopens)
>
>     adding file changes message
>
>     7 imports (3 dlopens)
>
>     added ... changesets message
>
>     4 imports (0 dlopens)
>
>     run 'hg update' message
>
> (I missed a "searching" message in there somewhere.)
>
> That's a total of 133 imports, 23 of which were satisfied by loading
> an extension module. The imports break down as follows:
>
> 51 imports (4 dlopens) from the mercurial package
> 5 imports from the hgext package
> 7 imports from the encodings package
>
> Everything else is imported from the top level, and at a glance
> appears to be all Python stdlib stuff.  The key period of execution
> looks to be between the printing of the Python banner and the printing
> of the adding changesets message. I see 46 imports (2 dlopens) from
> the mercurial or hgext packages. That leaves 40 stdlib imports, of
> which 16 were satisfied by dlopen.
>
> As a final check, I saved all the stdlib import statements from the -v
> output (77 statements) to a file and timed its execution:
>
>     % time /usr/bin/python ~/tmp/pyimp.py
>
>     real    0m0.162s
>     user    0m0.034s
>     sys     0m0.010s
>
> It doesn't take much time to import every stdlib module that Mercurial
> needs.  I don't know how much slower all this import machinery is in
> 3.x (and can't test at work, as we don't have a copy laying about). It
> would probably have to be 3x or more slower for it to have much
> visible impact on even simple hg commands.  I find it hard to believe
> that freezing the stdlib is going to lower the barrier enough for the
> Mercurial folks, if, in fact, import slowness is their main reason for
> not moving to 3.x.
>
> Skip
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com