Restricting import file lookup for pyd, dll, ...

[Bernard Lebel]

Hello,

It's been almost two months since I last investigated this issue, so
now I wish to revive this conversation.

To answer some of the questions raised by contributors....


[Gabriel Genellina] Try to shorten the PYTHONPATH to the really
required directories
(deleting those locations "that don't make sense").

[Bernard] I customized the PYTHONPATH using a pth file. The pth file
contains this:
\\Linuxserver\ANIMATION\XSI\WORKGROUP_ANIMATION\Data\Scripts
\\Linuxserver\ANIMATION\DB\MT\MT_WORKGROUP\Data\Scripts
\\Linuxserver\ANIMATION\DB\TS\TS_WORKGROUP\Data\Scripts
\\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE

That's it. I could hardly shorten these paths, unless the ressources
exposed through these paths were copied locally on every computer. Atm
I do not have management tools at my disposal to handle such a setup.

When print the paths:

C:\WINDOWS\system32\python24.zip
C:\Documents and Settings\blebel
C:\Python24\DLLs
C:\Python24\lib
C:\Python24\lib\plat-win
C:\Python24\lib\lib-tk
C:\Python24
d:\bernard\work\workgroups\workgroup_animation\data\scripts
d:\bernard\work\workgroups\mt_workgroup\data\scripts
d:\bernard\work\workgroups\ts_workgroup\data\scripts
\\Linuxserver\ANIMATION\FARM\PYTHON\RELEASE
c:\users\blebel\Softimage\XSI_5.11\Data\Scripts
C:\Python24\lib\site-packages
C:\Python24\lib\site-packages\win32
C:\Python24\lib\site-packages\win32\lib
C:\Python24\lib\site-packages\Pythonwin

If by "shortening the PYTHONPATH" you meant having less paths, I
hardly see how I could do less than what I do with the pth. All these
paths seem to be built in.

Now what would be REALLY nice is to have the ability to specify in the
paths the location of *specific files*. Is this possible? What happens
is that Python is visiting all kinds of locations to find some files,
like os, ntpath, and many more. If I could tell Python right away
where are these files located, I hope I could gain something.




[Fredrik Lundh] a plain Python 2.4 interpreter can start, execute a
command, and shut
down in about 0.13 seconds on my machine.  2.5 does the same thing in
0.10 seconds.

[Bernard] The problem is not firing up the standalone Python
interpreter. The problem is firing the application that embeds a
Python interpreter.

As explained above, when I start this application, many files are
looked in many different places. All PYTHONPATH locations are
traversed, where pyd-dll-py-pyw-pyc files are searched. Many of these
files are searched in highly improbable locations.




[Fredrik Lundh] are you sure you're benchmarking *Python's* start up
time, and not the
time it takes to load all the modules used by your application, or the
time it takes for "filemon" to print all those 4500 requests to the
monitor window?

[Bernard] The Filemon overhead consistently clocks at 1 second, not
matter the test I perform. Since I ran all tests with Filemon running,
I feel safe to ignore this constant.

Now, as to the first sentence, to be fair, no, I'm not exactly sure. I
don't know all the details of the Python's embedding in this software,
and this knowledge is out of my reach atm.

Here is the relevant Filemon log:
http://www.bernardlebel.com/img_remote/3D/XSI/python/Filemon_XSIPython.LOG
(710k file)



[Magnus Lycka] Sounds like a broken (networked?) file system. The only time I've
had that kind of problems with python startup is when I've had really
slow anti-virus programs that scanned all the files I opened. But then
it wasn't file requests that mattered, but actually opening them...
It still wasn't anywhere near 9 seconds though...

[Bernard] This is not entirely impossible, but I get similar results
on every computer I perform this test. By "every computer", I mean
different locations, with different hardwares and different network
setups. Even when the files are all local on the computer I get this.



Thanks
Bernard



On 10/28/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
> Magnus Lycka wrote:
>
> >> That's because I'm using Python through another application, via the
> >> pywin32 extensions. When that other application starts, it performs
> >> several thousands of file requests (we're talking 4,500, roughly) in
> >> the Python installation, locations where there are Python files, and
> >> in some other locations that don't make sense. This adds considerable
> >> time to the startup time of the application, we're talking between 2
> >> and 9 seconds.
> >
> > Sounds like a broken (networked?) file system. The only time I've
> > had that kind of problems with python startup is when I've had really
> > slow anti-virus programs that scanned all the files I opened. But then
> > it wasn't file requests that mattered, but actually opening them...
> > It still wasn't anywhere near 9 seconds though...
>
> if anyone finds out more about this issue, feel free to add a note to
> this FAQ entry:
>
> http://www.effbot.org/pyfaq/why-does-python-sometimes-take-so-long-to-start.htm
>
> </F>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>