[Patches] [ python-Patches-921466 ] Reduce number of open calls on startup

[SourceForge.net]

Patches item #921466, was opened at 2004-03-22 16:10
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=921466&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core (C code)
Group: None
Status: Open
Resolution: Accepted
Priority: 5
Submitted By: Martin v. Löwis (loewis)
Assigned to: Georg Brandl (gbrandl)
Summary: Reduce number of open calls on startup

Initial Comment:
This patch uses sys.path_importer_cache to reduce the
number of open calls, in the following way:
- if the value in path_importer_cache is None, it stats
the path to find out whether the file exists
- it then puts True/False into path_importer_cache
- if the value in path_importer_cache is False, the
path entry is skipped on all imports
- if the value is True, the stat call is skipped, and
open calls for files in the directory are made.

On Linux, this reduces the number of open calls for an
empty script from 343 to 263. The startup-time (for 100
interpreter invocations) goes down by one percent (from
0.0819s to 0.08113s per invocation).

----------------------------------------------------------------------

>Comment By: Neal Norwitz (nnorwitz)
Date: 2006-05-26 20:48

Message:
Logged In: YES 
user_id=33168

Georg, didn't you check this in or was that a diff patch?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-05-26 10:35

Message:
Logged In: YES 
user_id=21627

Your revised patch looks fine to me, so please apply.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2006-05-25 23:38

Message:
Logged In: YES 
user_id=33168

Without looking at the patch impl, I'm +1 on the idea of
reducing stat/open calls.  On NFS this is a huge time sync.

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-05-25 11:47

Message:
Logged In: YES 
user_id=849994

I reviewed this patch, in in consequence discovered a
problem with the sys.path_hooks machinery, described in
http://mail.python.org/pipermail/python-dev/2006-May/065173.html

This patch fixes the problem and corrects the original patch
to not set any sys.path_importer_cache entry to True or
False when no import hooks are enabled (the p_loader
argument to find_module is NULL then).

----------------------------------------------------------------------

Comment By: Georg Brandl (gbrandl)
Date: 2006-02-25 11:55

Message:
Logged In: YES 
user_id=849994

I'm very much for it. I haven't got too much RAM, and
whenever I start a Python program (emerge being the most
prominent example) after having worked heavily with e.g.
graphics or VMware, I'm hit by the files Python's opening
not being in the file cache anymore.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2006-02-20 13:51

Message:
Logged In: YES 
user_id=21627

Not sure. Anybody speaking in favour? against?

----------------------------------------------------------------------

Comment By: Georg Brandl (birkenfeld)
Date: 2006-02-20 02:42

Message:
Logged In: YES 
user_id=1188172

Can this go into 2.5?

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2004-03-23 07:43

Message:
Logged In: YES 
user_id=21627

It's certainly the case that the system has cached all files needed for 
startup in memory, including the directory contents of all directories 
searched.

OTOH, I assume that is the scenario in which people worry about startup 
time: high-frequency invocations of python. For a single invocation, it 
shouldn't matter much whether it takes 0.04s or 0.08s.

----------------------------------------------------------------------

Comment By: Raymond Hettinger (rhettinger)
Date: 2004-03-22 23:30

Message:
Logged In: YES 
user_id=80475

I am surprised that making 25% fewer open calls doesn't save
more than 1% in startup time.

One other thought, I wonder if the timing of these changes
is affected by the OS keeping recently loaded files in
buffers so that disk access time not included.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=921466&group_id=5470