[Python-Dev] new imputil.py

[Greg Stein]

Excellent... thanx for the feedback!

Comments:

On Mon, 3 Jan 2000, M.-A. Lemburg wrote:
>...
> The new importer does load everything in the test set
> (top level modules, packages, extensions within packages)
> without problems on Linux.

Great!

> Some comments:
>=20
> =B7 Why is the sys.path.insert(0,imputil.BuiltinImporter())=20
> needed in order to get b/w compatibility ?

Because I didn't want to build too much knowledge into the ImportManager.
Heck, I think adding sys.path removed some of the design elegence; adding
real knowledge of builtins... well, we'll just not talk about that. :-)

We could certainly do it this way; let's see what Guido says. I'm not
truly adverse to it, but I'd recommend against adding a knowledge of
BuiltinImporter to the ImportManager.

> =B7 Why is there no __path__ aware code in imputil.py (this is
> definitely needed in order to make it a drop-in replacement) ?

Because I don't like __path__ :-)  I don't think it would be too hard to
add, though.

If Guido says we need __path__, then I'll add it. I do believe there was a
poll a while back where he asked whether anybody truly used it. I don't
remember the result and/or Guido's resolution of the matter.

> =B7 Performance is still 50% of the Python builtin importer --
> a bummer if you ask me. More aggressive caching is definitely
> needed, perhaps even some recoding of methods in C.

I'm scared of caching and the possibility for false positives/negatives.

But yes, it is still slower and could use some analysis and/or recoding
*if* the speed is a problem. Slower imports does not necessarily mean they
are "too slow."

> =B7 The old chaining code should be moved into a subclass of
> its own.

Good thought. But really: I'd just rather torch it. This kind of depends
on whether we can get away with saying the ImportManager is *the* gateway
between the interpreter and Python-level import hooks. In other words,
will ImportManager be the *only* Python code to ever be allowed to call
sys.set_import_hook() ? If the ImportManager doesn't have to "play with
other import hooks", then the chaining can be removed altogether.

> =B7 The code should not import strop directly as this module
> will probably go away RSN. Use string methods instead.

Yah. But I'm running this against 1.5.2 :-)

I might be able to do something where the string methods are used if
available, and use the strop module if not.
[ similar to the 'os' bootstrapping that is done ]

Finn Bock emailed me to say that JPython does not have strop, but does
have string methods.

> =B7 The design of the ImportManager has some minor flaws: the
> FS importer should be settable via class attributes,

The class or the object itself? Putting a class in there would be nice, or
possibly passing it to the constructor (with a suitable default).

This is a good idea, though. Please clarify what you'd like to see, and
I'll get it added.

> deinstallation
> should be possible,

Maybe. This is somewhat dependent upon whether it must "play nice."
Deinstallation would be quite easy if we move to a sys.get/set style of
interface, and it wouldn't be an issue to do de-install code.

> a query mechanism to find the importer
> used by a certain import would also be nice to be able to
> verify correct setup.

module.__importer__ provides the importer that was used. This is defined
behavior (the system relies on that being set to deal with packages
properly).

Is this sufficient, or were you looking for something else?

module.__ispkg__ is also set to 0/1 accordingly.

For backwards compat, __file__ and __path__ are also set. The __all__
attribute in an __init__.py file is used for "from package import *".

> =B7 py/pyc/pyo file piping hooks would be nice to allow
> imports of signed (and trusted) code and/or encrypted code
> (a mixin class for these filters would do the trick).

I'd happily accept a base SuffixImporter class for these "pipes". I don't
believe that the ImportManager, Importer, or SuffixImporter base classes
would need any changes, though.

Note that I probably will rearrange the _fs_import() and friends, per
Guido's suggestion to move them into a base class. That may be a step
towards having "pipes" available.

> =B7 Wish list: a distutils importer hooked to a list of standard
> package repositories, a module to file location mapper to
> speed up file system based imports,=20

I'm not sure what the former would do. distutils is still a little
nebulous to me right now.

For a mapper, we can definitely have a custom Importer that knows where
certain modules are found. However, I suspect you're looking for some kind
of a cache, but there isn't a hook to say "I found <foo> at <this>
location" (which would be used to build the mapping).

Suggestions on both of these would be most welcome!

Cheers,
-g

--=20
Greg Stein, http://www.lyra.org/