[Python-Dev] Import redesign [LONG]

[Guido van Rossum]

Here's the promised response to Greg's response to my wishlist.

> On Thu, 18 Nov 1999, Guido van Rossum wrote:
> > Gordon McMillan wrote:
> >...
> > > I think imputil's emulation of the builtin importer is more of a 
> > > demonstration than a serious implementation. As for speed, it 
> > > depends on the test. 
> > 
> > Agreed.  I like some of imputil's features, but I think the API
> > need to be redesigned.
> 
> It what ways? It sounds like you've applied some thought. Do you have any
> concrete ideas yet, or "just a feeling" :-)  I'm working through some
> changes from JimA right now, and would welcome other suggestions. I think
> there may be some outstanding stuff from MAL, but I'm not sure (Marc?)

I actually think that the way the PVM (Python VM) calls the importer
ought to be changed.  Assigning to __builtin__.__import__ is a crock.
The API for __import__ is a crock.

> >...
> > So here's a challenge: redesign the import API from scratch.
> 
> I would suggest starting with imputil and altering as necessary. I'll use
> that viewpoint below.
> 
> > Let me start with some requirements.
> > 
> > Compatibility issues:
> > ---------------------
> > 
> > - the core API may be incompatible, as long as compatibility layers
> > can be provided in pure Python
> 
> Which APIs are you referring to? The "imp" module? The C functions? The
> __import__ and reload builtins?

> I'm guessing some of imp, the two builtins, and only one or two C
> functions.

All of those.

> > - support for rexec functionality
> 
> No problem. I can think of a number of ways to do this.

Agreed, I think that imputil can do this.

> > - support for freeze functionality
> 
> No problem. A function in "imp" must be exposed to Python to support this
> within the imputil framework.

Agreed.  It currently exports init_frozen() which is about the right
functionality.

> > - load .py/.pyc/.pyo files and shared libraries from files
> 
> No problem. Again, a function is needed for platform-specific loading of
> shared libraries.

Is it useful to expose the platform differences?  The current
imp.load_dynamic() should suffice.

> > - support for packages
> 
> No problem. Demo's in current imputil.
> 
> > - sys.path and sys.modules should still exist; sys.path might
> > have a slightly different meaning
> 
> I would suggest that both retain their *exact* meaning. We introduce
> sys.importers -- a list of importers to check, in sequence. The first
> importer on that list uses sys.path to look for and load modules. The
> second importer loads builtins and frozen code (i.e. modules not on
> sys.path).

This is looking like the redesign I was looking for.  (Note that
imputil's current chaining is not good since it's impossible to remove
or reorder importers, which I think is a required feature; an explicit
list would solve this.)

Actually, the order is the other way around, but by now you should
know that.  It makes sense to have separate ones for builtin and
frozen modules -- these have nothing in common.

There's another issue, which isn't directly addressed by imputil,
although with clever use of inheritance it might be doable.  I'd like
more support for this however.  Quite orthogonally to the issue of
having separate importers, I might want to recognize new extensions.
Take the example of the ILU folks.  They want to be able to drop a
file "foo.isl" in any directory on sys.path and have the ILU stubber
automatically run if you try to import foo (the client stubs) or
foo__skel (the server skeleton).

This doesn't fit in the sys.importers strategy, because they want to
be able to drop their .isl files in any directory along sys.path.
(Or, more likely, they want to have control over where in sys.modules
the directory/directories with .isl files are placed.)  This requires
an ugly modification to the _fs_import() function.  (Which should have
been a method, by the way, to make overriding it in a subclass of
PathImporter easier!)

I've been thinking here along the lines of a strategy where the
standard importer (the one that walks sys.path) has a set of hooks
that define various things it could look for, e.g. .py files, .pyc
files, .so or .dll files.  This list of hooks could be changed to
support looking for .isl files.

There's an old, subtle issue that could be solved through this as
well: whether or not a .pyc file without a .py file should be accepted
or not.  Long ago (in Python 0.9.8) a .pyc file alone would never be
loaded.  This was changed at the request of a small but vocal minority
of Python developers who wanted to distribute .pyc files without .py
files.  It has occasionally caused frustration because sometimes
developers move .py files around but forget to remove the .pyc files,
and then the .pyc file is silently picked up if it occurs on sys.path
earlier than where the .py was moved to.

Having a set of hooks for various extensions would make it possible to
have a default where lone .pyc files are ignored, but where one can
insert a .pyc importer in the list of hooks that does the right thing
here.  (Of course, it may be possible that this whole feature of lone
.pyc files should be replaced since the same need is easily taken care
of by zip importers.

I also want to support (Jim A notwithstanding :-) a feature whereby
different things besides directories can live on sys.path, as long as
they are strings -- these could be added from the PYTHONPATH env
variable.  Every piece of code that I've ever seen that uses sys.path
doesn't care if a directory named in sys.path doesn't exist -- it may
try to stat various files in it, which also don't exist, and as far as
it is concerned that is just an indication that the requested module
doesn't live there.

Again, we would have to dissect imputil to support various hooks that
deal with different kind of entities in sys.path.  The default hook
list would consist of a single item that interprets the name as a
directory name; other hooks could support zip files or URLs.  Jack's
"magic cookies" could also be supported nicely through such a
mechanism.

> Users can insert/append new importers or alter sys.path as before.
> 
> sys.modules continues to record name:module mappings.

Yes.

Note that the interpretation of __file__ could be problematic.  To
what value do you set __file__ for a module loaded from a zip archive?

> > - $PYTHONPATH and $PYTHONHOME should still be supported
> 
> No problem.
> 
> > (I wouldn't mind a splitting up of importdl.c into several
> > platform-specific files, one of which is chosen by the configure
> > script; but that's a bit of a separate issue.)
> 
> Easy enough. The standard importer can select the appropriate
> platform-specific module/function to perform the load. i.e. these can move
> to Modules/ and be split into a module-per-platform.

Again: what's the advantage of exposing the platform specificity?

> > New features:
> > -------------
> > 
> > - Integrated support for Greg Ward's distribution utilities (i.e. a
> >   module prepared by the distutil tools should install painlessly)
> 
> I don't know the specific requirements/functionality that would be
> required here (does Greg? :-), but I can't imagine any problem with this.

Probably more support is required from the other end: once it's common
for modules to be imported from zip files, the distutil code needs to
support the creation and installation of such zip files.  Also, there
is a need for the install phase of distutil to communicate the
location of the zip file to the Python installation.

> > - Good support for prospective authors of "all-in-one" packaging tool
> >   authors like Gordon McMillan's win32 installer or /F's squish.  (But
> >   I *don't* require backwards compatibility for existing tools.)
> 
> Um. *No* problem. :-)

:-)

> > - Standard import from zip or jar files, in two ways:
> > 
> >   (1) an entry on sys.path can be a zip/jar file instead of a directory;
> >       its contents will be searched for modules or packages

Note that this is what I mention above for distutil support.

> While this could easily be done, I might argue against it. Old
> apps/modules that process sys.path might get confused.

Above I argued that this shouldn't be a problem.

> If compatibility is not an issue, then "No problem."
> 
> An alternative would be an Importer instance added to sys.importers that
> is configured for a specific archive (in other words, don't add the zip
> file to sys.path, add ZipImporter(file) to sys.importers).

This would be harder for distutil: where does Python get the initial
list of importers?

> Another alternative is an Importer that looks at a "sys.py_archives" list.
> Or an Importer that has a py_archives instance attribute.

OK, but again distutil needs to be able to add to this list when it
installs a package.  (Note that package deinstallation should also be
supported!)

(Of course I don't require this to affect Python processes that are
already running; but it should be possible to easily change the
default search path for all newly started instances of a given Python
installation.)

> >   (2) a file in a directory that's on sys.path can be a zip/jar file;
> >       its contents will be considered as a package (note that this is
> >       different from (1)!)
> 
> No problem. This will slow things down, as a stat() for *.zip and/or *.jar
> must be done, in addition to *.py, *.pyc, and *.pyo.

Fine, this is where the caching comes in handy.

> >   I don't particularly care about supporting all zip compression
> >   schemes; if Java gets away with only supporting gzip compression
> >   in jar files, so can we.
> 
> I presume we would support whatever zlib gives us, and no more.

That's it. :-)

> > - Easy ways to subclass or augment the import mechanism along
> >   different dimensions.  For example, while none of the following
> >   features should be part of the core implementation, it should be
> >   easy to add any or all:
> > 
> >   - support for a new compression scheme to the zip importer
> 
> Presuming ZipImporter is a class (derived from Importer), then this
> ability is wholly dependent upon the author of ZipImporter providing the
> hook.

Agreed.  But since we're likely going to provide this as a standandard
feature, we must ensure that it provides this hook.

> The Importer class is already designed for subclassing (and its interface 
> is very narrow, which means delegation is also *very* easy; see
> imputil.FuncImporter).

But maybe it's *too* narrow; some of the hooks I suggest above seem to
require extra interfaces -- at least in some of the subclasses of the
Importer base class.

Note: I looked at the doc string for get_code() and I don't understand
what the difference is between the modname and fqname arguments.  If I
write "import foo.bar", what are modname and fqname?  Why are both
present?  Also, while you claim that the API is narrow, the multiple
return values (also the different types for the second item) make it
complicated.

> >   - support for a new archive format, e.g. tar
> 
> A cakewalk. Gordon, JimA, and myself each have archive formats. :-)
> 
> >   - a hook to import from URLs or other data sources (e.g. a
> >     "module server" imported in CORBA) (this needn't be supported
> >     through $PYTHONPATH though)
> 
> No problem at all.
> 
> >   - a hook that imports from compressed .py or .pyc/.pyo files
> 
> No problem at all.
> 
> >   - a hook to auto-generate .py files from other filename
> >     extensions (as currently implemented by ILU)
> 
> No problem at all.

See above -- I think this should be more integrated with sys.path than
you are thinking of.  The more I think about it, the more I see that
the problem is that for you, the importer that uses sys.path is a
final subclass of Importer (i.e. it is itself not further subclassed).
Several of the hooks I want seem to require additional hooks in the
PathImporter rather than new importers.

> >   - a cache for file locations in directories/archives, to improve
> >     startup time
> 
> No problem at all.
> 
> >   - a completely different source of imported modules, e.g. for an
> >     embedded system or PalmOS (which has no traditional filesystem)
> 
> No problem at all.
> 
> In each of the above cases, the Importer.get_code() method just needs to
> grab the byte codes from the XYZ data source. That data source can be
> cmopressed, across a network, on-the-fly generated, or whatever. Each
> importer can certainly create a cache based on its concept of "location".
> In some cases, that would be a mapping from module name to filesystem
> path, or to a URL, or to a compiled-in, frozen module.

See above for sys.path integration remark.

> > - Note that different kinds of hooks should (ideally, and within
> >   reason) properly combine, as follows: if I write a hook to recognize
> >   .spam files and automatically translate them into .py files, and you
> >   write a hook to support a new archive format, then if both hooks are
> >   installed together, it should be possible to find a .spam file in an
> >   archive and do the right thing, without any extra action.  Right?
> 
> Ack. Very, very difficult.

Actually, I take most of this back.  Importers that deal with new
extension types often have to go through a file system to transform
their data to .py files, and this is just too complicated.  However it
would be still nice if there was code sharing between the code that
looks for .py and .pyc files in a zip archive and the code that does
the same in a filesystem.  Hm, maybe even that shouldn't be necessary,
the zip file probably should contain only .pyc files...

(Unrelated remark: I should really try to release the set of modules
we've written here at CNRI to deal with zip files.  Unfortunately zip
files are hairy and so is our code.)

> The imputil scheme combines the concept of locating/loading into one step.
> There is only one "hook" in the imputil system. Its semantic is "map this
> name to a code/module object and return it; if you don't have it, then
> return None."

That's fine.  I actually don't recall where the find-then-load API
came from, I think it may be an artefact of the original
implementation strategy.  It is currently used as follows: we try to
see if there's a .pyc and then we try to see if there's a .py; if both
exist we compare the timestamps etc. to choose which one.  But that's
still a red herring.

> Your compositing example is based on the capabilities of the
> find-then-load paradigm of the existing "ihooks.py". One module finds
> something (foo.spam) and the other module loads it (by generating a .py).

I still don't understand why ihooks.py had to be so complicated.  I
guess I just had much less of an understanding of the issues.  (It was
also partly a compromise with an alternative design by Ken Manheimer,
who basically forced me to support packages, originally through ni.py.)

> All is not lost, however. I can easily envision the get_code() hook as
> allowing any kind of return type. If it isn't a code or module object,
> then another hook is called to transform it.
> [ actually, I'd design it similarly: a *series* of hooks would be called
>   until somebody transforms the foo.spam into a code/module object. ]

OK.  This could be a feature of a subclass of Importer.

> The compositing would be limited ony by the (Python-based) Importer
> classes. For example, my ZipImporter might expect to zip up .pyc files
> *only*. Obviously, you would want to alter this to support zipping any
> file, then use the suffic to determine what to do at unzip time.
> 
> > - It should be possible to write hooks in C/C++ as well as Python
> 
> Use FuncImporter to delegate to an extension module.

Maybe not so great, since it sounds like the C code can't benefit from
any of the infrastructure that imputil offers.  I'm not sure about
this one though.

> This is one of the benefits of imputil's single/narrow interface.

Plus its vague specs? :-)

> > - Applications embedding Python may supply their own implementations,
> >   default search path, etc., but don't have to if they want to piggyback
> >   on an existing Python installation (even though the latter is
> >   fraught with risk, it's cheaper and easier to understand).
> 
> An application would have full control over the contents of sys.importers.
> 
> For a restricted execution app, it might install an Importer that loads
> files from *one* directory only which is configured from a specific
> Win32 Registry entry. That importer could also refuse to load shared
> modules. The BuiltinImporter would still be present (although the app
> would certainly omit all but the necessary builtins from the build).
> Frozen modules could be excluded.

Actually there's little reason to exclude frozen modules or any
.py/.pyc modules -- by definition, bytecode can't be dangerous.  It's
the builtins and extensions that need to be censored.

We currently do this by subclassing ihooks, where we mask the test for
builtins with a comparison to a predefined list of names.

> > Implementation:
> > ---------------
> > 
> > - There must clearly be some code in C that can import certain
> >   essential modules (to solve the chicken-or-egg problem), but I don't
> >   mind if the majority of the implementation is written in Python.
> >   Using Python makes it easy to subclass.
> 
> I posited once before that the cost of import is mostly I/O rather than
> CPU, so using Python should not be an issue. MAL demonstrated that a good
> design for the Importer classes is also required. Based on this, I'm a
> *strong* advocate of moving as much as possible into Python (to get
> Python's ease-of-coding with little relative cost).

Agreed.  However, how do you explain the slowdown (from 9 to 13
seconds I recall) though?  Are you a lousy coder? :-)

> The (core) C code should be able to search a path for a module and import
> it. It does not require dynamic loading or packages. This will be used to
> import exceptions.py, then imputil.py, then site.py.

It does, however, need to import builtin modules.  imputil currently
imports imp, sys, strop and __builtin__, struct and marshal; note that
struct can easily be a dynamic loadable module, and so could strop in
theory.  (Note that strop will be unnecessary in 1.6 if you use string
methods.)

I don't think that this chicken-or-egg problem is particularly
problematic though.

> The platform-specific module that perform dynamic-loading must be a
> statically linked module (in Modules/ ... it doesn't have to be in the
> Python/ directory).

See earlier comments.

> site.py can complete the bootstrap by setting up sys.importers with the
> appropriate Importer instances (this is where an application can define
> its own policy). sys.path was initially set by the import.c bootstrap code
> (from the compiled-in path and environment variables).

I thing that algorithm (currently in getpath.c / getpathp.c) might
also be moved to Python code -- imported frozen.  Sadly, rebuilding
with a new version of a frozen module might be more complicated than
rebuilding with a new version of a C module, but writing and
maintaining this code in Python would be *sooooooo* much easier that I
think it's worth it.

> Note that imputil.py would not install any hooks when it is loaded. That
> is up to site.py. This implies the core C code will import a total of
> three modules using its builtin system. After that, the imputil mechanism
> would be importing everything (site.py would .install() an Importer which
> then takes over the __import__ hook).

(Three not counting the builtin modules.)

> Further note that the "import" Python statement could be simplified to use
> only the hook. However, this would require the core importer to inject
> some module names into the imputil module's namespace (since it couldn't
> use an import statement until a hook was installed). While this
> simplification is "neat", it complicates the run-time system (the import
> statement is broken until a hook is installed).

Same chicken-or-egg.  We can be pragmatic.

For a developer, I'd like a bit of robustness (all this makes it
rather hard to debug a broken imputil, and that's a fair amount of
code!).

> Therefore, the core C code must also support importing builtins. "sys" and
> "imp" are needed by imputil to bootstrap.
> 
> The core importer should not need to deal with dynamic-load modules.

Same question.  Since that all has to be coded in C anyway, why not?

> To support frozen apps, the core importer would need to support loading
> the three modules as frozen modules.

I'd like to see a description of how someone like Jim A would build a
single-file application using the new mechanism.  This could
completely replace freeze.  (Freeze currently requires a C compiler;
that's bad.)

> The builtin/frozen importing would be exposed thru "imp" for use by
> imputil for future imports. imputil would load and use the (builtin)
> platform-specific module to do dynamic-load imports.

Sure.

> > - In order to support importing from zip/jar files using compression,
> >   we'd at least need the zlib extension module and hence libz itself,
> >   which may not be available everywhere.
> 
> Yes. I don't see this as a requirement, though. We wouldn't start to use
> these by default, would we? Or insist on zlib being present? I see this as
> more along the lines of "we have provided a standardized Importer to do
> this, *provided* you have zlib support."

Agreed.  Zlib support is easy to get, but there are probably platforms
where it's not.  (E.g. maybe the Mac?  I suppose that on the Mac,
there would be some importer classes to import from a resource fork.)

> > - I suppose that the bootstrap is solved using a mechanism very
> >   similar to what freeze currently used (other solutions seem to be
> >   platform dependent).
> 
> The bootstrap that I outlined above could be done in C code. The import
> code would be stripped down dramatically because you'll drop package
> support and dynamic loading.

Not the dynamic loading.  But yes the package support.

> Alternatively, you could probably do the path-scanning in Python and
> freeze that into the interpreter. Personally, I don't like this idea as it
> would not buy you much at all (it would still need to return to C for
> accessing a number of scanning functions and module importing funcs).
> 
> > - I also want to still support importing *everything* from the
> >   filesystem, if only for development.  (It's hard enough to deal with
> >   the fact that exceptions.py is needed during Py_Initialize();
> >   I want to be able to hack on the import code written in Python
> >   without having to rebuild the executable all the time.
> 
> My outline above does not freeze anything. Everything resides in the
> filesystem. The C code merely needs a path-scanning loop and functions to
> import .py*, builtin, and frozen types of modules.

Good.  Though I think there's also a need for freezing everything.
And when we go the route of the zip archive, the zip archive handling
code needs to be somewhere -- frozen seems to be a reasonable choice.

> If somebody nukes their imputil.py or site.py, then they return to Python
> 1.4 behavior where the core interpreter uses a path for importing (i.e. no
> packages). They lose dynamically-loaded module support.

But if the path guessing is also done by site.py (as I propose) the
path will probably be wrong.  A warning should be printed.

> > Let's first complete the requirements gathering.  Are these
> > requirements reasonable?  Will they make an implementation too
> > complex?  Am I missing anything?
> 
> I'm not a fan of the compositing due to it requiring a change to semantics
> that I believe are very useful and very clean. However, I outlined a
> possible, clean solution to do that (a secondary set of hooks for
> transforming get_code() return values).

As you may see from my responses, I'm a big fan of having several
different sets of hooks.  I do withdraw the composition requirement
though.

> The requirements are otherwise reasonable to me, as I see that they can
> all be readily solved (i.e. they aren't burdensome).
> 
> While this email may be long, I do not believe the resulting system would
> be complex. From the user-visible side of things, nothing would be
> changed. sys.path is still present and operates as before. They *do* have
> new functionality they can grow into, though (sys.importers). The
> underlying C code is simplified, and the platform-specific dynamic-load
> stuff can be distributed to distinct modules, as needed
> (e.g. BeOS/dynloadmodule.c and PC/dynloadmodule.c).
> 
> > Finally, to what extent does this impact the desire for dealing
> > differently with the Python bytecode compiler (e.g. supporting
> > optimizers written in Python)?  And does it affect the desire to
> > implement the read-eval-print loop (the >>> prompt) in Python?
> 
> If the three startup files require byte-compilation, then you could have
> some issues (i.e. the byte-compiler must be present).

Another chicken-or-egg.  No biggie.

> Once you hit site.py, you have a "full" environment and can easily detect
> and import a read-eval-print loop module (i.e. why return to Python? just 
> start things up right there).

You mean "why return to C?"  I agree.  It would be cool if somehow
IDLE and Pythonwin would also be bootstrapped using the same
mechanisms.  (This would also solve the question "which interactive
environment am I using?" that some modules and apps want to see
answered because they need to do things differently when run under
IDLE,for example.)

> site.py can also install new optimizers as desired, a new Python-based
> parser or compiler, or whatever...  If Python is built without a parser or
> compiler (I hope that's an option!), then the three startup modules would
> simply be frozen into the executable.

More power to hooks!

--Guido van Rossum (home page: http://www.python.org/~guido/)