Tons of stats/opens to non-existing files increases Python's startupon loaded NFS servers

[Dieter Maurer]

"Fredrik Lundh" <fredrik at pythonware.com> writes on Fri, 6 May 2005 00:08:36 +0200:
> > ... lots of "no such file or directory ...
> > Whoa!! After looking at what is being stat'd or
> > open'd, it looks like 'encodings' is new in 2.4 and,
> > even "worse", everything is looked for as a zip first.
> 
> so why not build that ZIP?

We, too, saw this problem -- and we had the "*.zip" files already...

Python is a bit stupid in its import logic.
When, e.g., a package "P" defined in a zip archive "zzz.zip"
contains an "import os", then Python checks whether
"zzz.zip" contains "P.os" (that is okay). *But*, usually
"zzz.zip" does not define "P.os" (as "os" is a builtin module)
and then Python checks in the file system
for "zzz.zip/P/os{,.py,pyc,.so,} and "zzz.zip/P/osmodule.so".
Of course, all of them fail as "zzz.zip" is a Zip archive
and "zzz.zip/something" is not meaningfull as a file system reference.

I improved on this by patching Python's "import.c" with the
attached patch. The patch implements that a "path_hook"
declaring itself to be responsible for a path is authoritative
for both negative as well as positive "find_module" responses.
Earlier, a negative "find_module" response caused Python to
try the default module lookup.


Furthermore, it is vital that your "sys.path" is as small as possible
because a single module lookup can cause file system lookups
in the order of 4 times the number of path elements. 

The standard extension of "sys.path" often contains far more
path elements than necessary (if you defined "python24.zip", you
should remove all other python library directories that do not contain
shared objects).

Dieter

-------------- next part --------------
A non-text attachment was scrubbed...
Name: import.pat
Type: text/x-path
Size: 817 bytes
Desc: Patch for Python 2.4.1 to prevent meaningless fs checks	for zip archive members
URL: <http://mail.python.org/pipermail/python-list/attachments/20050506/962beca5/attachment.bin>