[Python-Dev] PEP 420 - dynamic path computation is missing rationale

[Eric V. Smith]

On 05/21/2012 07:25 PM, Nick Coghlan wrote:
> As a simple example to back up PJE's explanation, consider:
> 1.  encodings becomes a namespace package
> 2. It sometimes gets imported during interpreter startup to initialise
> the standard io streams
> 3. An application modifies sys.path after startup and wants to
> contribute additional encodings
> 
> Searching the entire parent path for new portions on every import would
> be needlessly slow.
> 
> Not recognising new portions would be needlessly confusing for users. In
> our simple case above, the application would fail if the io
> initialisation accessed the encodings package, but work if it did not
> (e.g. when all streams are utf-8).
> 
> PEP 420 splits the difference via an automatically invalidated cache:
> when you iterate over a namespace package __path__ object, it rescans
> the parent path for new portions *if and only if* the contents of the
> parent path have changed since the previous scan.

That seems like a pretty convincing example to me.

Personally I'm +1 on putting dynamic computation into the PEP, at least
for top-level namespace packages, and probably for all namespace packages.

The code is not very large or complicated, and with the proposed removal
of the restriction that sys.path cannot be replaced, I think it behaves
well.

But Guido can decide against it without hurting my feelings.

Eric.

P.S.: Here's the current code in the pep-420 branch. This code still has
the restriction that sys.path (or parent_path in general) can't be
replaced. I'll fix that if we decide to keep the feature.

class _NamespacePath:
    def __init__(self, name, path, parent_path, path_finder):
        self._name = name
        self._path = path
        self._parent_path = parent_path
        self._last_parent_path = tuple(parent_path)
        self._path_finder = path_finder

    def _recalculate(self):
        # If _parent_path has changed, recalculate _path
        parent_path = tuple(self._parent_path)     # Make a copy
        if parent_path != self._last_parent_path:
            loader, new_path = self._path_finder(self._name, parent_path)
            # Note that no changes are made if a loader is returned, but we
            #  do remember the new parent path
            if loader is None:
                self._path = new_path
            self._last_parent_path = parent_path   # Save the copy
        return self._path

    def __iter__(self):
        return iter(self._recalculate())

    def __len__(self):
        return len(self._recalculate())

    def __repr__(self):
        return "_NamespacePath" + repr((self._path, self._parent_path))

    def __contains__(self, item):
        return item in self._recalculate()