[Python-Dev] Re: .DLL vs .PYD search order

[Greg Stein]

On Sat, 4 Dec 1999, M.-A. Lemburg wrote:
> Fredrik Lundh wrote:
>...
> > sorry, I still don't understand it.  our applications already
> > use different storage mechanisms, databases, signatures,
> > lazy importing, version handling, etc, etc.  now, if *we*
> > have managed to build all that on top of an old version
> > of imputil.py, how come it's not sufficient for the rest
> > of you?
> 
> I've tried to get (an older) imputil.py version up and running
> too. It did work, but only after some considerable tweaking
> and even with integrated cache mechanisms did not reach
> the performance of the builtin importer (which doesn't
> use the kinds of caching strategies I had built into
> imputil.py).

1) yes, it was an older version and did not have the PathImporter class.
   As a by product, the DirectoryImporters that it *did* have were much
   slower. It still did not support builtins, frozen modules, or dynamic
   loads. All of that is present now, so it works "out of the box" much
   better.

2) Performance: as I wrote in the other email, I don't believe that is an
   argument against the design. The imputil approach *will* be slower than
   the current Python mechanism, but there is some more coding to do to
   truly see how much. The side benefits (e.g. ZipImporter and caching)
   may outweigh the result. Time will tell.

> Getting the whole setup to work wasn't easy
> at all, because of the way imputil importers delegate work
> and things get even more confusing when it starts to "take
> over" certain parts of packages by installing temselves
> as importers for a particular package.

I don't understand this. If it is relevant, then please expand. Thx.

> > > A chain of simple minded importers won't work together
> > > too well
> > 
> > why?  it sure works for us...
> 
> An example: 
> 
> A path importer knows how to scan directories and how to use
> a path to tell the correct order. It can maybe also import
> .py/.pyc/.pyo files. Now what happens if it finds a shared
> lib as module... the usual imputil way would be to delegate
> the request to some other importer which can handle shared
> libs... but wait: how does the shared lib importer know
> where to look ? It will have to rescan the directories,
> etc...

No, the "usual imputil way" is that the PathImporter understands searching
a path and loading stuff from that path. An Importer is a combination of
locating and loading (since they are, typically, tightly bound). The next
rev will allow user-plugging of support for new file types.

> > > duplicate work
> > 
> > avoiding duplicate work is what object oriented design
> > is all about.  and last time I checked, Python had excellent
> > support for that.
> 
> See my example above.
> 
> The agent approach used by imputil does not support
> OO design too well: even though you can avoid duplicate
> programming work on the importers by using a few
> base classes which implement dir scans, shared lib
> imports, etc. the imputil design does not provide
> means to avoid duplicate actions taken by the importers.

There is always a balance to be struck between independence and coupling.
I chose to reduce coupling and increase independence. If you shift a bunch
of stuff out of the Importers, then you will increase the coupling between
the imputil framework and the Importers. That coupling will then close off
future possibilities.

Within the framework itself (e.g. between _import_hook and get_code),
there is a lot of opportunity for change. Since that is behind the covers,
it is no big deal to shift functionality around. I plan to do so.

>...
> Looks like you are in ranting mode here ;-) Seriously,
> I've checked my imputil.py version (with caches enabled)
> against the builtin importer and noticed a performance
> downgrade by factor >2. This was enough to convince me
> of looking for other techniques to handle the problems
> I had at the time... you know, relative imports and things.

I have run a long series of tests. Without doing any performance work on
imputil, the ratio is 9 to 13. The 13 may have bumped up to about 15 or 16
when I added some dynamic loading code (I forget). Regardless, it is
definitely less than a 2X increase. And that is with zero optimization.

*shrug*

I'm done. I'll do some code in a couple weeks.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/