[Python-ideas] Refactoring the import system to be object-oriented at the top level

[Brett Cannon]

On Sat, Nov 7, 2009 at 20:24, Nick Coghlan <ncoghlan at gmail.com> wrote:
> (Disclaimer: this is complicated Py-in-the-sky stuff, and I'm handwaving
> away a lot of major problems with the concept, not least of which is the
> sheer amount of work involved. I just wanted to get the idea published
> somewhere while I was thinking about it)
>
> I'm in the process of implementing a runpy.run_path function for 2.7/3.2
> to allow Python code to use the zipfile and directory execution feature
> provided by the CPython command line in 2.6/3.1. It turns out the global
> state used for the import system is causing some major pain in the
> implementation. It's solvable, but it will probably involve a couple of
> rather ugly hacks and the result sure as hell isn't going to be thread-safe.
>
> Anyway, the gist of the idea in the subject line is to take all of the
> PEP 302 data stores and make them attributes of an ImportEngine class.
> This would affect at least:
>
> sys.modules
> sys.path
> sys.path_hooks
> sys.path_importer_cache
> sys.meta_path
> sys.dont_write_bytecode
>
> The underlying import machinery would migrate to instance methods of the
> new class.
>

Do you really mean methods or just instance attributes? I personally
don't care personally, but it does require more of an API design
otherwise.

> The standard import engine instance would be stored in a new sys module
> attribute (e.g. 'sys.import_engine'). For backwards compatibility, the
> existing sys attributes would remain as references to the relevant
> instance attributes of the standard engine.

How would that work? Because they are module attributes there is no
way to use a property to have them return what the current
sys.import_engine uses.

>
> Modules would get a new special attribute (e.g. '__import_engine__')
> identifying the import engine that was used to import that module.
> __import__ would be modified to take the new special attribute into account.
>

Take into account how? As in when importing a package to always use
the import engine used for the parent module in the package?

> The main immediate benefit from my point of view would be to allow runpy
> to create *copies* of the standard import engine so that
> runpy.run_module and runpy.run_path could go do their thing without
> impacting the rest of the interpreter. At the moment that really isn't
> feasible, hence the lack of thread safety in that module.
>
> I suspect such a change would greatly simplify experimentation with
> Python security models as well: restricted code could be given a
> restricted import engine rather than the restrictions having to be baked
> in to the standard import engine.
>

Huh, I wonder made you think about that example? =)

> >From an OO design point of view, it's a classic migration of global
> state and multiple functions to manipulate that state into a single
> global instance of a new class.

If anything it makes it easier to discover everything that affects
importing instead of having to crawl through the sys docs to find
every attribute that happens to mention the word "import".

-Brett