[Python-Dev] Re: .DLL vs .PYD search order

[M.-A. Lemburg]

Fredrik Lundh wrote:
> 
> M.-A. Lemburg <mal@lemburg.com> wrote:
> > >     given a name, check if you have a
> > >     module with that name, and install
> > >     it.  if you cannot, fail (in which case
> > >     python asks the next importer along
> > >     the path).
> > >
> > > why do you have to complicate things beyond that?
> > > why not just let Python provide a few base classes
> > > and mixins for people who want to create custom
> > > importers, and be done with it?
> >
> > Because importing in Python has become *much* more
> > complicated over time. There are requests for new
> > features which touch subjects such as storage mechanisms,
> > lookups, signatures (for trusted code), lazy imports, etc.
> 
> sorry, I still don't understand it.  our applications already
> use different storage mechanisms, databases, signatures,
> lazy importing, version handling, etc, etc.  now, if *we*
> have managed to build all that on top of an old version
> of imputil.py, how come it's not sufficient for the rest
> of you?

I've tried to get (an older) imputil.py version up and running
too. It did work, but only after some considerable tweaking
and even with integrated cache mechanisms did not reach
the performance of the builtin importer (which doesn't
use the kinds of caching strategies I had built into
imputil.py). Getting the whole setup to work wasn't easy
at all, because of the way imputil importers delegate work
and things get even more confusing when it starts to "take
over" certain parts of packages by installing temselves
as importers for a particular package.
 
> > A chain of simple minded importers won't work together
> > too well
> 
> why?  it sure works for us...

An example: 

A path importer knows how to scan directories and how to use
a path to tell the correct order. It can maybe also import
.py/.pyc/.pyo files. Now what happens if it finds a shared
lib as module... the usual imputil way would be to delegate
the request to some other importer which can handle shared
libs... but wait: how does the shared lib importer know
where to look ? It will have to rescan the directories,
etc...
 
> > duplicate work
> 
> avoiding duplicate work is what object oriented design
> is all about.  and last time I checked, Python had excellent
> support for that.

See my example above.

The agent approach used by imputil does not support
OO design too well: even though you can avoid duplicate
programming work on the importers by using a few
base classes which implement dir scans, shared lib
imports, etc. the imputil design does not provide
means to avoid duplicate actions taken by the importers.

> > and downgrade performance considerably due to the
> > many recursive function calls
> 
> now that's what I call premature optimization.  and this
> scares the hell out of me: if the rest of the python-dev
> crowd don't seriously believe that Python is (or can be
> made) fast enough to implement things like this, why
> the heck are you using Python at all?  am I the only
> one here who doesn't believe in osterhout's talk about
> "the great system vs. scripting language divide"?

Looks like you are in ranting mode here ;-) Seriously,
I've checked my imputil.py version (with caches enabled)
against the builtin importer and noticed a performance
downgrade by factor >2. This was enough to convince me
of looking for other techniques to handle the problems
I had at the time... you know, relative imports and things.

-- 
Marc-Andre Lemburg
______________________________________________________________________
Y2000:                                                    27 days left
Business:                                      http://www.lemburg.com/
Python Pages:                           http://www.lemburg.com/python/