[Python-Dev] Proposal: expose PEP 302 facilities via 'imp' and 'pkgutil'

[Phillip J. Eby]

I just noticed that two stdlib modules (runpy and test.test_importhooks) 
contain reimplementations of the base PEP 302 algorithm, or loaders 
wrapping the standard (pre-302) import machinery.

Meanwhile, the 'imp' module exports an undocumented IMP_HOOK constant 
(since Python 2.3), that is used internally to Python/import.c but never 
actually returned from the imp API.

Further, the machinery called by imp.find_module() actually does the full 
PEP 302 search dance - but then skips any PEP 302 importers in the process, 
because the wrapper doesn't let it return a loader object.

What I'd like to do is make the necessary modifications to import.c that 
would allow you to access the loaders found by the C version of find_module.

I propose to create a new API, 'imp.find_loader()' and have it return a PEP 
302-compatible loader object, even for cases that would normally be handled 
via 'imp.load_module()'.  In such cases, the loader returned would be an 
instance of one of a loader class similar to those in runpy, 
test_importhooks, and setuptools (which also has similar code).

What I'm not sure of is where to put the loader class.  It seems to me 
there should be a stdlib module, but it doesn't seem worth writing in C, 
especially with so many Python implementations floating around.

I could create a new Python module for them, but we already have so many 
import-related modules floating around.  Would it be reasonable to add them 
to 'pkgutil', which until now has contained only one function?  This would 
help cut down on some of the code duplication, without adding yet another 
module to the stdlib.

An additional issue: "pydoc" needs to be able to determine what submodules, 
if any, exist in a package, but the PEP 302 protocol does not provide for 
this ability.  I'd like to add optional additional methods to the PEP 302 
"importer" protocol (and to any stdlib importer objects) to support the 
type of filesystem-like queries performed by pydoc.  This should allow 
pydoc to be compatible with zip files as well as regular files, and any 
future PEP 302 importers that provide the necessary features.

Having these features would also let me cut some code out of setuptools' 
"pkg_resources" module, that adds some of these features using adapter 
registries.

It may be too late for me to be able to implement all of this in time for 
alpha 2, but at minimum I think the 'find_loader()' addition and the move 
of the import wrapper classes to 'pkgutil' could be accomplished.

Replacing pydoc's myriad stat(), listdir() and other path hacking with 
clean importer protocols might be out of reach, but I think it's worth a 
try.  For one thing, it would mean that shipping the Python stdlib in a 
zipfile would not inhibit pydoc's ability to display help for stdlib 
modules and packages.  (Currently, pydoc does not work correctly with 
packages that are in zipfiles or which are spread across multiple 
directories, and it is unable to "discover" modules that are in zipfiles.)