[Python-Dev] import too slow on NFS based systems

[PJ Eby]

On Thu, Jun 21, 2012 at 10:08 AM, Daniel Braniss <danny at cs.huji.ac.il>wrote:

> > On Thu, 21 Jun 2012 13:17:01 +0300
> > Daniel Braniss <danny at cs.huji.ac.il> wrote:
> > > Hi,
> > > when lib/python/site-packages/ is accessed via NFS, open/stat/access
> is very
> > > expensive/slow.
> > >
> > > A simple solution is to use an in memory directory search/hash, so I
> was
> > > wondering if this has been concidered in the past, if not, and I come
> > > with a working solution for Unix (at least Linux/Freebsd) will it be
> > > concidered.
> >
> > There is such a thing in Python 3.3, although some stat() calls are
> > still necessary to know whether the directory caches are fresh.
> > Can you give it a try and provide some feedback?
>
> WOW!
> with a sample python program:
>
> in 2.7 there are:
>        stats   open
>        2736    9037
> in 3.3
>        288     57
>
> now I have to fix my 2.7 to work with 3.3 :-)
>
> any chance that this can be backported to 2.7?
>

As Antoine says, not in the official release.  You can, however, speed
things up substantially in 2.x by zipping the standard library and placing
it in the location given in the default sys.path, e.g.:

# python2.7
Python 2.7 (r27:82500, May  5 2011, 11:50:25)
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> [p for p in sys.path if p.endswith('.zip')]
['/usr/lib/python27.zip']

If you include a compiled 'sitecustomize.py' in this zipfile, you would
also be able to implement a caching importer based on the default one in
pkgutil, to take up the rest of the slack.  I've previously posted sketches
of such importers; they're not that complicated to implement.  It's just
that if you don't *also* zip up the standard library, your raw interpreter
start time won't get much benefit.

(To be clear, creating the zipfile will only speed up stdlib imports,
nothing else; you'll need to implement a caching importer to get any
benefit for site-packages imports.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120621/86b84045/attachment.html>