[Python-Dev] PEP 420 - dynamic path computation is missing rationale

[Eric V. Smith]

On 05/23/2012 03:56 PM, Brett Cannon wrote:
> 
> 
> On Wed, May 23, 2012 at 3:35 PM, PJ Eby <pje at telecommunity.com
> <mailto:pje at telecommunity.com>> wrote:
> 
>     On Wed, May 23, 2012 at 3:02 PM, Brett Cannon <brett at python.org
>     <mailto:brett at python.org>> wrote:
> 
>         If I understand the proposal correctly, this would be a change
>         in NamespaceLoader in how it sets __path__ and in no way affect
>         any other code since __import__() just grabs the object on
>         __path__ and passes as an argument to the meta path finders
>         which just iterate over the object, so I have no objections to it.
> 
> 
>     That's not *quite* the proposal (but almost).  The change would also
>     mean that __import__() instead passes a ModulePath (aka Nick's
>     LazyIterable) instance to the meta path finders, which just iterate
>     over it.  But other than that, yes.
> 
> 
> And why does __import__() need to construct that? I thought
> NamespaceLoader was going to be making these "magical" __path__ objects
> that detected changes and thus update themselves as necessary and just
> stick them on the object. Why specifically does __import__() need to
> play a role? 

Assume that we're talking about importing either a top-level namespace
package named 'parent' and a nested namespace package parent.child.

The problem is that NamespaceLoader is just passed the parent path
(typically sys.path, but if a sub-package then parent.__path__). The
concern is that if the parent path object is replaced:
sys.path = sys.path + ['new-dir']
or
parent.__path__ = ['new-dir']
then the NamespaceLoader instance can no longer detect changes to
parent_path.

So the proposed solution is for NamespaceLoader to be told the name of
the parent module ('sys' or 'parent') and the attribute name to use to
find the path ('path' or '__path__').

Here's another suggestion: instead of modifying the finder/loader code
to pass these names through, assume that we can always find
(module_name, attribute_name) with this code:

def find_parent_path_names(module):
    parent, dot, me = module.__name__.rpartition('.')
    if dot == '':
        return 'sys', 'path'
    return parent, '__path__'

>>> import glob
>>> find_parent_path_names(glob)
('sys', 'path')
>>> import unittest.test.test_case
>>> find_parent_path_names(unittest.test.test_case)
('unittest.test', '__path__')

I guess it's a little more fragile than passing in these names to
NamespaceLoader, but it requires less code to change.

I think I'll whip this up in the pep-420 branch.

Eric.