[Python-Dev] PEP 451 update

[Brett Cannon]

On Fri, Oct 25, 2013 at 2:10 PM, PJ Eby <pje at telecommunity.com> wrote:

> On Fri, Oct 25, 2013 at 1:15 PM, Brett Cannon <brett at python.org> wrote:
> >
> > On Fri, Oct 25, 2013 at 12:24 PM, PJ Eby <pje at telecommunity.com> wrote:
> >> At least through all 2.x, reload() just uses module.__name__ to
> >> restart the module find-and-load process, and does not assume that
> >> __loader__ is valid in advance.
> >
> >
> > That doesn't make much sense in a post-importlib world where import makes
> > sure that __loader__ is set (which it has since Python 3.3). Otherwise
> you
> > are asking for not just a reload but a re-find as well.
>
> That's a feature, not a bug.  A reload() after changing sys.path
> *should* take into account the change, not to mention any changes to
> meta_path, path hooks, etc.  (And it's how reload() worked before
> importlib.)
>

Fair enough, but in my mind that obviously doesn't really click for what I
view as a reload in an importlib world where secret import code no longer
exists. When I think re-load I think "load again", not "find the module
again and execute a load with a possibly new loader".

It's not worth changing in Python 3.3, but if you want to propose to
redefine importlib.reload() to use importlib.find_spec() to reset the
__spec__ on a module then I'm not going to argue against it (but I'm not
going to fight for it either).

And in a PEP 451 world it should be dead-simple to make this work the way
you want in your own code even if this doesn't go the way you want::

  spec = importlib.find_spec(name)
  module.__spec__ = spec
  importlib.reload(module)  # Which in itself is essentially
init_module_attrs(spec, module); spec.loader.exec_module(module)

Heck, you can do this in Python 3.3 right now::

  loader = importlib.find_loader(name)
  module = sys.modules[name]
  module.__loader__ = loader
  importlib.reload(module)


> I suppose it's not really documented all that well,


Nope =)


> but way way back
> in the 2.3 timeframe I asked for a tweak to PEP 302 to make sure that
> reload() (in the re-find sense) would work properly with PEP 302
> loaders like zipimport -- some of the language still in the PEP is
> there specifically to support this use case.  (Specifically, the bit
> that says loaders *must* use the existing module object in sys.modules
> if there's one there already, so that reload() will work.  It was
> actually in part to ensure that reload() would work in the case of a
> re-find.)
>

Ah, okay. That is not explicit in the PEP beyond coming off a total
nuisance in order to support reloading by the loader, not an explicit
finder + loader use-case.


>
> It appears that since then, the PEP has been changed in a way that
> invalidates part of the purpose of the prior change; I guess I missed
> the discussion of that change last year.  :-(
>
> ISTM there should've been such a discussion, since IIRC importlib
> wasn't supposed to change any Python semantics, and this is a
> non-trivial change to the semantics of reload() even in cases that
> aren't doing lazy imports or other such munging.


Well, not change them to the best of its abilities. I'm sure there are edge
cases that I couldn't match properly since I could only go by the test
suite and bug reports filed since Python 3.1.


>  reload() used to
> take sys.* and __path__ changes into account, and IMO should continue
> to do so.


Unfortunately this was never explicitly documented (at least that I
noticed) nor was there a test in the stdlib to enforce compliance with it,
else this never would have happened.


>  If this is an intentional change in reload() semantics,
> other Python implementations need to know about this too!
>

Not really. As of Python 3.3 the semantic shift included re-implementing
the function in pure Python so they pick up the change for free; remember
the function is not a builtin anymore to somewhat de-emphasize its use
since it's kind of a hack and really only used typically at the interpreter
prompt as an easy part of a load-edit-reload dev cycle, not some
fundamental operation.


>
> (That being said, I'm not saying I shouldn't or couldn't have tested
> this in 3.3 and found out about it that way.  And the existence of
> issue18698 suggests that nobody's relying yet on even the
> *fundamental* semantics of PEP 302 reload() working properly in 3.3,
> since that was an even bigger change that nobody spotted till a couple
> of months ago.)
>

Yep, never got a bug report on this (although there have been reports of
typos in the docs as well as not returning what's in sys.modules so it is
being used by people).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20131025/5f93172e/attachment.html>