[Import-SIG] Import engine PEP up on python.org

["Martin v. Löwis"]

> It's desirable for the same reason *any* form of object-oriented
> encapsulation is desirable: because it makes it easier to *think*
> about the problem and manage interdependencies between state updates.

I guess I'm -1 on that PEP then. If it introduces complications just
for the sake of some presumed simplification, it's not worth it.

> I didn't realise the merits of OO designs needed to be justified - I
> figured the list of 6 pieces of interdependent process global state
> spoke for itself.

Perhaps I'm challenging the specific choice of classes then: I would
find it completely reasonable to move all of this into the interpreter
state, as I think it's fine that this "global" state is unique to the
interpreter. There is only a single __import__ builtin, and the
objective of the import statement is to make a change to the "global"
state (scoped with the interpreter).

>> I notice that there is overlap both with multiple subinterpreters,
>> and with restricted execution. It hints at providing both, but actually
>> provides neither.
> 
> It doesn't claim to provide either - it's sole aim is to provide a
> relatively lightweight mechanism to selectively adjust elements of the
> import system (e.g. adding to sys.path when importing plugins, but
> leaving it alone otherwise).

Ok - that might be a use case. However, I'm skeptical that this PEP is
good at achieving that objective - as you notice, there is the challenge
of recursive imports.

I would rather prefer to make such variables per-thread, or, rather
"per context". Something like

with sys.extended_path(directory):
  load_plugin()

This would extend the path for all code run within the context of the
with statement, but not elsewhere. As an implementation strategy, the
thread state would be able to override the global variables, in a
stacked (nested) manner. The exact list of variables that can be
overridden needs to be carefully considered - for example, I would
still view a single sys.modules as important in that use case.


> But having the import state better encapsulated would make it easier
> to improve the isolation of subinterpreters so that they aren't
> sharing Python modules, even if they still share extension modules

That already works, no? Subinterpreters don't share Python modules
(that's about the only feature about subinterpreters that actually
works).

> No, they're not. Yes, the hooks are *usable*, but they're damn hard to
> comprehend. When even the *experts* hate messing with a subsystem,
> it's a hint that there's something wrong with the way it is set up. In
> this case, I firmly believe a big part of the problem is that the
> import system is a complex, interdependent mechanism, but there's no
> coherence to the architecture. It's as if the whole thing was written
> in C from an architectural point of view, but without even bothering
> to create a dedicated structure to hold the state.

I agree that the import system is difficult to understand, and the
PEP 302 hooks in particular. I mightily disagree that the cause of
these difficulties is the global state used in the implementation.
It's rather the order in which things are called, and how they interact,
which makes it difficult to understand. Adding an optional keyword
argument to some of the function is surely no simplification.

Regards,
Martin