[Import-SIG] Thoughts on cleaner reloading support

[Eric Snow]

On Mon, Sep 2, 2013 at 7:53 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> The extension module discussion on python-dev got me thinking about
> the different ways in which the "singleton" assumption for modules can
> be broken, and how to ensure that extension modules play nicely in
> that environment.
>
> As I see it, there are 4 ways the "singleton that survives for the
> lifetime of the process following initial import" assumption regarding
> modules can turn out to be wrong:
>
> 1. In-place reload, overwriting the existing contents of a namespace.
> imp.reload() does this. We sort of do it for __main__, except we
> usually keep re-using that namespace to run *different* things, rather
> than rerunning the same code.
>
> 2. Parallel loading. We remove the existing module from sys.modules
> (keeping a reference to it alive), and load a second copy.
> Alternatively, we call the loader APIs directly. Either way, we end up
> with two independent copies of the "same" module, potentially
> reflecting difference system states at the time of execution.
>
> 3. Subinterpreter support. Quite similar to parallel loading, but
> we're loading the second copy because we're in a subinterpreter and
> can't see the original.
>
> 4. Unloading. We remove the existing module from sys.modules and drop
> all other references to it. The module gets destroyed, and we later
> import a completely fresh copy.
>
> Even pure Python modules may not support these, since they may have
> side effects, or assume they're in the main interpreter, or other
> things. Currently, there is no way to signal this to the import
> system, so we're left with implicit misbehaviour when we attempt to
> reload the modules with global side effects.
>

This is a great summary.  It got me thinking big (from which I usually pare
my ideas down to something sane-ish <wink>).


>
> For a while, I was thinking we could design the import system to "just
> figure it out", but now I'm thinking a selection of read/write
> properties on spec objects may make more sense:
>
>     allow_reload
>     allow_unload
>     allow_reimport
>     allow_subinterpreter_import
>
> These would all default to True, but loaders and modules could
> selectively turn them off.
>

2 things:

1. These make more sense to me on the loader (though perhaps exposed on the
spec).
2. These (and other related attributes) may be easier to digest if bundled
into a LoaderCapabilities named tuple.

For these attributes to be useful to the import system we would have to
have several other module registries akin to sys.modules (or one registry
that manages the extra info).

Brainstorming on this I came up with an expanded list:

allow_raw (basically create_module() called directly)
allow_unregister (a.k.a. allow_unload)
shared_globally (the module should be shared across all subinterpreters)

allow_recreate (create again after unregister/unload)
allow_create_parallel
allow_exec_in_place
allow_reexec (create again after unregister/unload)
allow_exec_parallel

For the sake of global (across subinterpreters) module registries, the same
attributes could either be stored in a nested version of the named tuple or
as more attributes, like this:

allow_recreate_global
allow_create_parallel_global
allow_exec_in_place_global
allow_reexec_global
allow_exec_parallel_global

I also realized that ImportEngine/ImportSystem (PEP 406) has a lot of the
same trickiness as subinterpreters regarding reloading.


> They would also be advisory


Ah, here's where you were talking about "advisory" APIs. :)

rather than enforced via all possible
> import state manipulation mechanisms. New functions in importlib.util
> could provide easier alternatives to directly manipulating
> sys.modules:
>
> - importlib.util.reload (replacement for imp.reload that checks the
> spec allows reloading)
> - importlib.util.unload (replacement for "del
> sys.modules[module.__name__]" that checks the spec allows unloading,
> and also unloads all child modules)
> - importlib.util.reimport (replacement for
> test.support.import_fresh_module that checks the spec of any existing
> sys.module entry allows reimporting a parallel copy)
>

At moments like this I keep thinking about PEP 406... :)


>
> One of these is not like the others... aside from the existing
> extension module specific mechanism defined in PEP 3121, I'm not sure
> we can devise a general *loader* level API to force imports for a
> particular name to fail in a subinterpreter. So this concern probably
> needs to be ignored in favour of a possible future C API level
> solution.
>

Again, great write-up.  I think you nailed it.

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/import-sig/attachments/20130918/d7f41fa6/attachment.html>