Restricting import file lookup for pyd, dll, ...

[Bernard Lebel]

Oops, sorry for the inconsistency. The pth file rather looks like this:

d:\bernard\work\workgroups\workgroup_animation\data\scripts
d:\bernard\work\workgroups\mt_workgroup\data\scripts
d:\bernard\work\workgroups\ts_workgroup\data\scripts
\\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
c:\users\blebel\Softimage\XSI_5.11\Data\Scripts


Bernard



On 12/11/06, Bernard Lebel <3dbernard at gmail.com> wrote:
> Hello,
>
> It's been almost two months since I last investigated this issue, so
> now I wish to revive this conversation.
>
> To answer some of the questions raised by contributors....
>
>
> [Gabriel Genellina] Try to shorten the PYTHONPATH to the really
> required directories
> (deleting those locations "that don't make sense").
>
> [Bernard] I customized the PYTHONPATH using a pth file. The pth file
> contains this:
> \\Linuxserver\ANIMATION\XSI\WORKGROUP_ANIMATION\Data\Scripts
> \\Linuxserver\ANIMATION\DB\MT\MT_WORKGROUP\Data\Scripts
> \\Linuxserver\ANIMATION\DB\TS\TS_WORKGROUP\Data\Scripts
> \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
>
> That's it. I could hardly shorten these paths, unless the ressources
> exposed through these paths were copied locally on every computer. Atm
> I do not have management tools at my disposal to handle such a setup.
>
> When print the paths:
>
> C:\WINDOWS\system32\python24.zip
> C:\Documents and Settings\blebel
> C:\Python24\DLLs
> C:\Python24\lib
> C:\Python24\lib\plat-win
> C:\Python24\lib\lib-tk
> C:\Python24
> d:\bernard\work\workgroups\workgroup_animation\data\scripts
> d:\bernard\work\workgroups\mt_workgroup\data\scripts
> d:\bernard\work\workgroups\ts_workgroup\data\scripts
> \\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
> c:\users\blebel\Softimage\XSI_5.11\Data\Scripts
> C:\Python24\lib\site-packages
> C:\Python24\lib\site-packages\win32
> C:\Python24\lib\site-packages\win32\lib
> C:\Python24\lib\site-packages\Pythonwin
>
> If by "shortening the PYTHONPATH" you meant having less paths, I
> hardly see how I could do less than what I do with the pth. All these
> paths seem to be built in.
>
> Now what would be REALLY nice is to have the ability to specify in the
> paths the location of *specific files*. Is this possible? What happens
> is that Python is visiting all kinds of locations to find some files,
> like os, ntpath, and many more. If I could tell Python right away
> where are these files located, I hope I could gain something.
>
>
>
>
> [Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a
> command, and shut
> down in about 0.13 seconds on my machine.  2.5 does the same thing in
> 0.10 seconds.
>
> [Bernard] The problem is not firing up the standalone Python
> interpreter. The problem is firing the application that embeds a
> Python interpreter.
>
> As explained above, when I start this application, many files are
> looked in many different places. All PYTHONPATH locations are
> traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these
> files are searched in highly improbable locations.
>
>
>
>
> [Fredrik Lundh] are you sure you're benchmarking *Python's* start up
> time, and not the
> time it takes to load all the modules used by your application, or the
> time it takes for "filemon" to print all those 4500 requests to the
> monitor window?
>
> [Bernard] The Filemon overhead consistently clocks at 1 second, not
> matter the test I perform. Since I ran all tests with Filemon running,
> I feel safe to ignore this constant.
>
> Now, as to the first sentence, to be fair, no, I'm not exactly sure. I
> don't know all the details of the Python's embedding in this software,
> and this knowledge is out of my reach atm.
>
> Here is the relevant Filemon log:
> http://www.bernardlebel.com/img_remote/3D/XSI/python/Filemon_XSIPython.LOG
> (710k file)
>
>
>
> [Magnus Lycka] Sounds like a broken (networked?) file system. The only time I've
> had that kind of problems with python startup is when I've had really
> slow anti-virus programs that scanned all the files I opened. But then
> it wasn't file requests that mattered, but actually opening them...
> It still wasn't anywhere near 9 seconds though...
>
> [Bernard] This is not entirely impossible, but I get similar results
> on every computer I perform this test. By "every computer", I mean
> different locations, with different hardwares and different network
> setups. Even when the files are all local on the computer I get this.
>
>
>
> Thanks
> Bernard
>
>
>
> On 10/28/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
> > Magnus Lycka wrote:
> >
> > >> That's because I'm using Python through another application, via the
> > >> pywin32 extensions. When that other application starts, it performs
> > >> several thousands of file requests (we're talking 4,500, roughly) in
> > >> the Python installation, locations where there are Python files, and
> > >> in some other locations that don't make sense. This adds considerable
> > >> time to the startup time of the application, we're talking between 2
> > >> and 9 seconds.
> > >
> > > Sounds like a broken (networked?) file system. The only time I've
> > > had that kind of problems with python startup is when I've had really
> > > slow anti-virus programs that scanned all the files I opened. But then
> > > it wasn't file requests that mattered, but actually opening them...
> > > It still wasn't anywhere near 9 seconds though...
> >
> > if anyone finds out more about this issue, feel free to add a note to
> > this FAQ entry:
> >
> > http://www.effbot.org/pyfaq/why-does-python-sometimes-take-so-long-to-start.htm
> >
> > </F>
> >
> > --
> > http://mail.python.org/mailman/listinfo/python-list
> >
>