[Import-SIG] How best to replace imp.load_module()?

[Eric Snow]

On Apr 4, 2014 12:58 PM, "Brett Cannon" <bcannon at gmail.com> wrote:
>
> I've been thinking about what it takes to replace imp and I realized that
imp.load_module() is the hardest to replace for two reasons. One issue is
that importlib.abc.create_module() can -- and does -- return None. This
means that if someone wanted to replicate the
imp.find_module()/imp.load_module() dance in a PEP 451 world it takes::
>
>   spec = importlib.find_spec(name)
>   try:
>     module = spec.loader.create_module(spec)
>   except AttributeError:
>     module = None
>   if module is None:
>     module = types.ModuleType(spec.name)
>   # No clear way to set import-related attributes.
>   spec.loader.exec_module(module)
>
> It took 6 lines to get a module. That seems a bit excessive and ripe to
either have an importlib.util function that handles this all correctly or
simply make create_module() a required method on a loader and have
importlib.abc.create_module() return types.ModuleType(spec.name).

I agree with your later email about not needing to add a ton of API
unnecessarily.  I'm not sure imp.load_module() needs to live on in
importlib.

The key one I'd like to see is a replacement for direct calls to
loader.load_module().  We do this a bunch in the stdlib and each of those
places currently has to do a dance around _SpecMethods.  Doing so outside
the stdlib isn't really correct.  In both cases it would be nice to wrap
that in a util function (like "import_from_loader()") or a classmethod on
Loader.

>
> The second annoyance is that we have not exposed
_SpecMethod.init_module_attrs() in any way.

If we had an import_from_loader(), I'm not sure we'd need to worry about
it.  Would there be other use cases for setting those attrs?

Also, once I've wrapped up (either way) the OrderedDict-related stuff I'm
working on, my main goal is to propose a successor to PEP 406
(ImportEngine).  That would including exposing most of the _SpecMethods API
in some form.

FWIW, I'm still uncomfortable with exposing that API directly on
ModuleSpec, but would like to see it exposed publicly in some indirect way.

> That can either come through in importlib.util or we can make
types.ModuleType grow a method that takes a spec and then sets all the
appropriate attributes.

Maybe if it were a class-only method.  I'd hate to see something like that
exposed on module objects.

> If we go with the former we should make sure that in importlib we always
prefer __spec__ over any module-level values and that one can pass in a
spec to types.ModuleType to set __spec__ so that in the distant Python 4
future we can deprecate all module-level attributes and just work off of
__spec__.

This is tricky.  It depends on how useful it is to people to have module
attrs that vary from the spec (for which the module was originally loaded).

That aside, I've found it's still useful to have __name__ and __file__
rather than having to look them up on __spec__.  Maybe that's just because
I'm not used to it. :)  That would be a different matter if the common use
cases for the two were satisfied by other means.  It would be nice if we
could restrict __file__ to just modules that are FS-based (and drop it or
set it to None for all other modules).  The others attrs are probably used
uncommonly enough that they could get dropped from module objects.

>
> I think if we can get these two bits cleaned up we can tell people who
use imp.load_module() directly that they can::
>
>   # Assume proper loader chosen.
>   spec = importlib.util.spec_from_loader(loader)
>   module = some_newfangled_way_of_doing_this()
>   somehow_init_module_attrs(module)
>   loader.exec_module(module)

That's basically what import_from_loader() would do.

>
> which isn't that bad for something they probably should be avoiding as
much as possible to begin with.

Maybe it would be worth getting a clear idea of why people use
imp.load_module() and loader.load_module() (directly).  My guess is that it
is to accomplish slight deviations from normal import behavior.  For
example, that last timed I looked at the source, Salt used
imp.load_module() to do some trickery.  However, I expect that most cases
would be satisfied by use of proper importers.  Either way, it would be
nice to have a better picture of what unusual things people are doing like
this.  I'd benefit from that at least. :)

>
> Or we take the easiest option and simply ignore all of these issues and
just say that working outside of import is not something we want to worry
about in the stdlib and let PyPI come up with some utility code that does
all of this for you if you really want it. The code maintainer in me is
liking this idea + making it easier to set __spec__ on a module through its
constructor.

:)

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20140405/eba1fe0f/attachment.html>