[Import-SIG] Import engine PEP up on python.org

[Nick Coghlan]

On Sun, Nov 13, 2011 at 6:28 PM, "Martin v. Löwis" <martin at v.loewis.de> wrote:
> Am 13.11.2011 06:12, schrieb Nick Coghlan:
>> I finally got around to updating the import engine draft PEP and
>> publishing it on python.org: http://www.python.org/dev/peps/pep-0406/
>
> I think the rationale section needs to be improved. In fact, I still
> don't understand what the objective of this API is (I do understand what
> it achieves, but it's unclear why having that is desirable, and for what
> applications).

It's desirable for the same reason *any* form of object-oriented
encapsulation is desirable: because it makes it easier to *think*
about the problem and manage interdependencies between state updates.
I didn't realise the merits of OO designs needed to be justified - I
figured the list of 6 pieces of interdependent process global state
spoke for itself.

> I notice that there is overlap both with multiple subinterpreters,
> and with restricted execution. It hints at providing both, but actually
> provides neither.

It doesn't claim to provide either - it's sole aim is to provide a
relatively lightweight mechanism to selectively adjust elements of the
import system (e.g. adding to sys.path when importing plugins, but
leaving it alone otherwise).

But having the import state better encapsulated would make it easier
to improve the isolation of subinterpreters so that they aren't
sharing Python modules, even if they still share extension modules
(you could put a single pointer to the import engine on the
interpreter state rather than storing it in sys the way we do now).

> I think the long-term solution really should be proper support for
> subinterpreters, where there is no global state in C at all. Extension
> modules already can achieve this through the PEP 3121 API (even though
> few modules actually do).
>
> If the objective is to have more of the import machinery implemented in
> Python, then making importlib the import machinery might be best.

Guido already approved (in principle) that change - this PEP would
actually depend on that being done first (because I think trying to
build this API directly on top of import.c would be a complete waste
of time).

> If the objective is to allow hooks into the import procedure, it would
> be best to just provide the hooks. OTOH, PEP 302 already defined hooks,
> and it seems that people are happy with these.

No, they're not. Yes, the hooks are *usable*, but they're damn hard to
comprehend. When even the *experts* hate messing with a subsystem,
it's a hint that there's something wrong with the way it is set up. In
this case, I firmly believe a big part of the problem is that the
import system is a complex, interdependent mechanism, but there's no
coherence to the architecture. It's as if the whole thing was written
in C from an architectural point of view, but without even bothering
to create a dedicated structure to hold the state.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia